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SECTION  I.  Executive  Overview 


A.  Background 


Policy,  legislation,  and  regulations  have  been  issued  on  contractor  past  performance  as  it 
relates  to  Government  contracting  over  the  past  three  and  one  half  years. 

Methods  and  approaches  to  implement  this  direction  within  the  Department  of  Defense 
(DOD)  have  been  studied,  discussed,  and  evaluated  by  officials  in  the  Office  of  the 
Secretary  of  Defense  (OSD)  and  in  the  DOD  components.  The  Past  Performance 
Coordinating  Council  (PPCC)  has  also  been  active  in  developing  a  DOD-wide  position 
on  contractor  past  performance. 

The  implications  of  these  actions  has  caused  some  concern  within  the  DOD  as  to  the 
best  approach  for  implementing  the  contractor  past  performance  initiative.  Because  of 
these  concerns,  the  Deputy  Under  Secretary  of  Defense  (Acquisition  Reform) 
det^mined  that  an  independent  study  was  needed  before  implementing  a  department¬ 
wide  policy  dealing  with  matters  related  to  contractor  past  performance. 

The  cummt  plan  within  the  DOD  is  to  issue  a  change  to  the  Department  of  Defense 
FAR  Supplemoit  (DFARS)  to  clarify  how  contractor  past  performance  will  be  handled 
within  the  Department  of  Defense.  A  position  on  the  specie  content  of  the  DFARS 
change  is  in  the  final  stages  of  coordination. 

There  continues  to  be  some  concern  and  reservation  by  officials  within  the  DOD  about 
issues  that  surround  the  implementation  of  actions  currently  directed  by  the  existing 
policy  and  regulations.  Policy  and  regulations  state  that  implementing  action  can  be 
tailored  to  the  particular  circumstances  and  nature  of  a  procurement  program,  yet,  there 
has  been  no  consensus  on  the  extent  of  the  tailoring  that  can  and  should  be  done,  nor  on 
who  should  be  empowered  to  do  the  tailoring. 

The  Deputy  Under  Secretary  of  Defense  (Acquisition  Reform)  has  concluded  that 
further  study  is  needed  before  implementing  a  department- wide  policy  Healing  with 
matters  related  to  contractor  past  performance. 


B.  Objective  and  Scope 

The  principal  objective  of  the  study  was  to  provide  information  and  independent 
evaluations  that  will  assist  in  the  formulation  of  a  DOD-wide  policy  on  the  collection 
and  use  of  information  on  the  past  performance  of  contractors.  The  scope  of  the  study 
was  focused  in  two  broad  areas: 
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The  first  area  addressed  the  following  items: 

•  All  existing  past  performance  processes  and  systems  used  within  the  DOD,  and  a 
sample  of  comparable  processes  and  systems  used  by  other  federal  agencies  and 
by  commercial  firms; 

•  The  manner  in  which  past  performance  information  is  collected  and  validated; 

•  The  past  performance  data  elements  prescribed  by  the  functional  users; 

•  Customer  satisfaction  with  past  performance  information  and  the  systems  that 
provide  this  information; 

•  Customer  views  on  the  difference  that  past  performance  information  makes  in 
the  source  selection  process; 

•  An  economic  analysis  to  determine  whether  the  use  of  past  performance 
information  makes  good  business  sense; 

•  The  appropriate  use  of  contractor  past  performance  information; 

•  The  current  use  of  past  performance  information  within  DOD;  and 

•  The  administrative  burden  associated  with  collecting  the  information. 

The  second  broad  area  in  which  the  study  focused  involved  the  development  of  a  model 
program  to  assist  in  the  implementation  of  past  performance  policy  within  DOD.  This 
model  was  developed  using  the  results  from  an  assessment  of  die  items  listed  above, 
under  the  first  focus  area. 


C.  Summary  of  Results. 

Section  n  of  the  report  provides  a  detailed  discussion  of  the  study.  The  Executive 
Overview  summarizes  these  results  in  the  following  areas: 

•  Government  perspective  for  dealing  with  contractor  past  performance, 

•  Industry  perspective  for  dealing  with  contractor  past  performance, 

•  A  proposed  model  for  dealing  widi  contractor  past  performance  in  DOD,  and 

•  A  business  case  analysis  that  evaluates  some  of  the  major  alternatives  for  dealing 
with  contractor  past  performance  in  DOD. 

1.  Government  Perspective 

This  study  area  evaluated  existing  contractor  past  performance  processes  and  systems. 
A  prerequisite  to  this  evaluation  was  a  definition  of  terms  and  a  structure  for  organizing 
the  information  that  is  currently  available  and  relevant  to  the  past  performance  of 
contractors.  The  definitions  that  were  used  made  a  distinction  between  the  following 
types  of  information  that  pertains  to  the  past  performance  of  contractors: 

•  Performance  information  gathered  at  tiie  time  of  a  specific  procurement  decision 
on  an  ad  hoc  basis  and  for  the  exclusive  use  in  that  decision,  and 

•  Performance  information  gathered  on  a  routine,  basis  as  contract  work  is 
performed,  is  further  divided  into: 
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-  Information  gathered  for  purposes  of  managing  the  active  contracts,  and 

-  Information  gathered  with  the  intention  that  it  will  be  made  available  for 
use  in  acquisition  decisions  at  some  future  date. 

When  identifying  and  evaluating  existing  past  performance  information  systems,  focus 
was  placed  on  the  information  available  for  use  in  acquisition  decisions  at  some  future 
date.  The  following  criteria  were  also  used  to  precisely  identify  the  existing  processes 
and  systems: 

•  Information  is  collected,  validated  and  filed  to  support  future  source  selection 
decisions 

•  Opportunity  is  provided  for  review,  comment  and  rebuttal  of  information  by  the 
contractor 

•  Provisions  are  established  to  resolve  disputes  between  the  contractor  and  the 
government 

•  Information  is  subject  to  the  same  controls  and  safeguards  as  other  information 
used  in  source  selection  decisions 

•  The  system  is  in  operation  and  currently  supporting  source  selection  decisions. 


DOD  Past  Performance  Processes  and  Systems 

The  applicatioc  of  these  definitions  and  criteria  revealed  a  very  limited  coverage  for  past 
performance  information  systems  that  are  currently  in  use  in  DOD  relative  to  the 
requirements  «ivisioned  by  die  proposed  FAR  and  DFARS  implonentation. 

The  analysis  revealed  two  basically  different  types  of  systems.  One  type  relied  on  an 
appraisal  of  tire  contractor's  performance  by  an  offidal,  or  officials,  in  a  position  to 
make  a  judgment  on  how  well  the  contractor  had  performed  his  ta.<5k  This  is  the  system 
envisioned  by  tiie  new  Part  42. 15  of  the  FAR.  The  other  system  relied  on  quality  and 
delivery  information  gathered  and  recorded  for  the  purpose  of  traddng  tire  specific  line 
items  delivered  under  the  terms  of  an  existing  contract  A  tiiird  tystem  involved  the 
certification  of  contractors  based  on  their  past  performance  and  used  information  taken 
fi^om  the  systems  noted  above. 

Using  this  definition  structure,  the  following  contractor  past  performance  systems  were 
identified: 

•  Performance  appraisal  systems  included: 

-  Contractor  Performance  Assessment  Reporting  System  (CPARS). 

Developed  and  used  within  the  Air  Force  primarily  for  maior  systems 
acquisition  programs. 

-  A&E  Contract  Administration  Support  System  (ACASS).  Developed  and 
used  by  the  Army  Corps  of  Engineers. 

-  Construction  Contract  Appraisal  Support  System  (CCASS).  Developed 
and  used  by  the  Army  Corps  of  Engineers. 
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•  Performance  tracking  systems  included: 

■  Red/Yellow/Green  (RYG)  system.  Developed  by  the  Navy  Supply 
Systems  Command  and  used  by  some  procurement  organizations  within 
the  Navy  when  procuring  commodities  with  an  assigned  Federal  Supply 
Classification  code. 

-  Automated  Best  Value  Model  (AB  VM).  Developed  and  used  within  DLA, 
also  when  procuring  commodities  with  an  assigned  Federal  Supply 
Classification  code. 

-  Contractor  Profile  System  (CPS).  Developed  by  DLA  and  which  has  seen 
some  limited  use. 

Section  11.  A.  of  the  report  discusses  and  evaluates  these  existing  systems  based  on 
experience  to  date. 


Other  Government  Departments  and  Agencies 

Past  performance  policy  implementation  was  also  a  matter  of  concern  within  the  other 
Government  agencies.  The  GSA  Federal  Supply  Service  was  found  to  have  an  existing 
system.  Most  other  Government  agendes  were  focused  on  collecting  and  validating 
past  performance  information  for  tiituie  use  in  procurement  decisions  (Le.,  the 
requirements  contained  in  the  new  Part  42.15  of  the  FAR). 

One  of  the  most  notewortiiy  findings  was  a  NASA  decision  not  to  collect  past 
p^formance  information  before  tiieir  contracts  were  signed.  This  decision  was  based  on 
the  fact  that  award  fee  contracts  cover  80%  of  NASA's  procurement  dollars.  These 
contracts  already  provide  for  a  periodic  evaluation  of  the  contractor's  performance. 
Another  factor  in  this  decision  was  a  recent  experience  with  a  contractor  evaluation 
system  that  proved  to  be  an  administrative  burden  and  that  did  not  provide  the  expected 
benefits.  NASA  will,  continue  to  use  past  performance  as  a  standard  evaluation  factor 
in  source  selection. 

Principal  Results 

From  a  Government  perspective,  the  study  revealed  the  following: 

•  The  DOD  acquisition  program  is  mammoth,  covers  multiple  business  areas,  and 
dwarfs  industrial  conglomerates  in  the  range  of  products  and  services  procured, 
the  number  of  procurement  organizations,  and  the  dollar  amount  of  the  program. 

•  When  used  as  an  evaluation  factor  in  source  selection,  past  performance 
information  is  typically  gathered  when  needed  and  on  an  ad  hoc  basis 

•  There  are  currently  very  few  instances  where  past  performance  information  is 
collected  and  validated  on  a  continuing  basis  for  future  use  in  contractor 
selection  decisions  (as  now  required  by  Part  42.15  of  the  FAR); 

•  In  all  cases  where  past  performance  information  is  collected  and  validated  for 
future  use,  the  process  and  related  system  were  tailored  to  a  specific  business 
area 

•  There  are  two  principal  approaches  for  collecting  past  performance  information 
on  a  continuing  basis  — 
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■  Performance  appraisals  (i.e.,  a  "report  card"  at  the  contract  or  delivery 
order  level),  and 

■  Performance  tracking  (quality  and  delivery  data  at  the  contract  line  item 
level) 


•  The  routine  collection  of  past  performance  information  has  been  tried 
penodically  over  the  past  thirty  years.  These  initiatives  have  typically  been 
abandoned  because  the  value  of  the  information  in  contracting  decisions  was  not 
found  to  offset  the  cost  and  administrative  burden  of  collecting  validating,  and 
maintaining  the  information. 


2.  Industry  Perspective 

The  use  of  contractor  past  performance  information  was  goierally  found  to  be  an 
integral  element  in  broader  programs  designed  to  improve  die  purchased  goods 
component  of  the  cost  of  goods  sold.  The  motivation  for  these  supplier  evaluation 
programs  was  provided  by  a  need  to  achieve  and  sustain  a  competitive  position  in  the 
market  place.  The  material  purchased  by  the  average  U.S.  manufacturing  finn  typically 
ranges  from  about  40%  to  65%  of  sales,  dius,  attrition  to  these  cost  elements,  to 
supplier  evaluation  programs,  and  to  suppli^  relationships,  was  found  to  be  of  major 
importance  to  the  continued  viability  of  the  industrial  operations. 

Another  tnraid  that  was  evident  in  many  industries,  was  a  move  to  ftstah|i<i?h  more 
profitable,  longer-term  relationships  with  fewer  suppliers.  An  important  factor  in  this 
process  was  the  demonstrated  performance  of  particular  suppliers  based  on  a  number  of 
criteria,  including  the  past  performance  of  the  suppliers. 

Althou^  the  industry  programs  varied  in  many  of  dieir  details,  one  of  the  common 
elements  was  a  rccogmtion  that  successful  programs  needed  to  be  tailored  to  discrete 
business  areas.  An  initial  step  in  this  process  was  a  thorou^  analysis  of  the  specific 
business  area  widi  regard  to  company  requirements,  past  and  projected,  as  well  as  to 
industry  trends  and  the  specific  contractors  and  suppliers  that  represented  current  and 
potential  sources  of  supply.  This  step  was  design^  to  lead  to  a  sensible  program 
depending  upon  the  specific  business  area. 

In  summary,  the  following  were  the  principal  results  from  an  industry  perspective: 

•  There  is  no  acquisition  program  anywhere  in  industry  that  is  comparable  to 
DOD's  program  in  size,  scope  and  conqrlexity 

•  All  past  performance  initiatives  were  focused  on  discrete  business  areas  or 
strategic  business  units 

•  The  motivation  for  past  performance  programs  in  industry  is  to  improve  or 
maintain  the  conq)etitive  position  of  the  operation 

•  All  programs  are  integrated  with  overall  strategic  and  planning  considerations 
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•  Industry  programs  tend  to  exhibit  common  characteristics  which  include: 

-  Business  area  strategy  and  management 

-  Supplier  approval 

-  Quality  system  assessment 

-  Perfonnance  assessmrat 

-  Performance  measurement  feedback 

-  Supplier  development  and  improvement  initiatives 

-  Total  cost  assessments 

-  Item  and  part-level  ceitification 

-  Recognition  programs 

A  discussion  of  industry  initiatives  and  each  of  die  program  characteristics  listed  above 
is  contained  in  Section  n  B  of  this  report 


3.  Contractor  Evaluation  Program  Model 

One  of  the  principal  challenges  in  conducting  the  study  was  to  find  ways  to  discuss  and 
explore  alternatives  for  dealing  with  contractor  past  performance  and  related  issues 
such  as  business  area  analyses,  strategic  planning,  contracting  for  best  value,  improving 
contractor  performance  levels,  and  gatiiaing  past  performance  information  A 
Contractor  Evaluation  Program  model  was  developed  to  serve  as  a  frame  of  tefiaence 
for  evolving  a  consensus  on  a  viable  ^)inoaclL  This  model  was  also  developed  to 
address  the  related  issues  identified  above. 

The  diagram,  on  tiie  next  page,  portrays  the  major  elements  of  the  model  functions  tiiat 
are  typically  involved  in  tire  acquisition  of  products  and  services.  In  the  cento*  of  tiie 
diagram  are  the  three  principal  elements  of  the  model.  The  left  side  of  the  diagram 
identifies  the  major  functions  that  deal  with  the  collection,  validation,  and  retortion  of 
past  performance  information  for  future  use  and  which  are  affected  by  the  outcome  of 
the  business  analysis,  the  business  area  strategy,  and  the  business  area  evaluatimi  plan. 
The  right  side  of  the  diagram  depicts  the  major  functions  typically  performed  doting 
contractor  selection. 

This  model  was  developed  fixrm  tire  Attiiur  D.  Little  case  histories  and  discussions  with 
Government  and  industry  offidals  who  represented  a  cross-section  of  the  functions  and 
disciplines  involved  in  tire  acquisition  of  products  and  services.  In  addition,  two 
workshops  were  held  with  representatives  of  Acquisition  Reform  Senior  Steering  Group 
(ARSSG)  members. 


A  description  of  the  model  is  contained  in  Section  H  C  of  the  report.  To  some  extent, 
the  model  may  be  viewed  as  a  "straw  man"  at  this  stage,  nevertheless  it  is  believed  to  be 
an  important  step  in  achieving  the  objectives  of  the  study.  The  Contractor  Evaluation 
Program  model  has  three  principal  elements,  as  depicted  below. 
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Develop  a  strategy  for  eacfi 
business  area  that  addresses  t^  e 
collection  and  use  of  contractor 
past  perfonnance  infonnatior^ 


Define  a  process  for  eacfT^ 
business  area  to  evaluate 
past  performance  and  to  use{ 
this  informalion  for  source 
s^^setection  purposes 


These  principal  elements  are: 

•  Business  Area  Plan  ~  provides  an  analysis  of  the  business  areas  in  which 
contracts  are  awarded, 

•  Business  Area  Strategy  —  establishes  goals  and  objectives  for  use  of  contractor 
past  performance  information  and  related  issues  in  tiie  bu^ess  area,  and 

•  Business  Area  Evaluation  Process  —  defines  the  process  for  executing  tiie  plan 
and  strategy  in  the  business  area. 

The  principles  used  in  designing  and  developing  tire  model  included  the  following: 

•  A  cost-effective  approach  to  the  collection  and  use  of  contractor  past 
performance  information  depends  on,  and  is  sensitive  to  factors  related  to  the 
business  areas  in  which  products  and  services  are  procured  and  used  (as  opposed 
to  a  universal  approach  that  can  be  applied  to  the  full  range  of  products  and 
services  procured  by  DOD  in  all  areas). 

•  A  business  area  consists  of  a  homogeneous  group  of  products  or  services  which 
share  similar  characteristics  and  for  which  a  forward-looldng  plan  and  a  coherent 
and  congruous  strategy  and  evaluation  process  can  be  developed. 

•  Business  areas  can  be  local  or  extended  in  application,  and  in  their  most  robust 
form,  they  constitute  the  horizontal  integration  of  products  and  services. 

•  The  process  for  implementing  contractor  past  performance  issues  in  a  particular 
business  area  is  developed  from  business  area  plans  and  strategies  for  the 
specific  business  area  and  typically  involves  a  cross-functional  team  effort. 

•  The  initial  and  vital  step  in  developing  plans  and  strategy  for  a  business  area  is 
an  analysis  that  covers  the  requirements  for  the  product  or  service,  past  and 
projected;  the  industry  composition  and  basis  of  competition,  and  the  market 
trends  and  performance  of  leading  companies  in  the  industry. 

•  The  business  area  plan  and  strategy  will  provide  the  basis  for  developing  a 
tailored  approach  to  the  collection  and  use  of  contractor  past  performance 
information  in  the  particular  business  area  as  well  as  the  foundation  for  a  total 
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program  designed  to  incorporate  best  value  practices  into  the  procurement 
process  and  to  attract  contractors  and  suppliers  committed  to  high  levels  of 
performance. 

•  Information  technology  will  be  utilized  to  facilitate  communication  between 
Government  managers  in  sq)arate  organizations  with  a  need  to  share  information 
about  business  area  strategies  and  plans  and  the  past  performance  of  contractors 
in  those  business  areas. 

A  description  of  the  Contractor  Evaluation  Program  is  contained  in  Section  n  C  of  the 
report 


4.  Business  Case  Analysis 

Because  this  study  is  designed  to  primarily  assist  in  the  implementation  of  past 
p^ormance  policy,  and  not  in  systems  development  the  business  case  anal)^  is 
therefore  focused  on  three  alternative  policy  implementation  approaches  to  past 
performance.  The  three  alternative  models  used  in  the  analysis  were  the  “As-is”  model, 
which  is  structured  from  the  information  in  Section  H  A;  the  DEARS  model  developed 
from  information  in  the  FAR  and  the  proposed  changes  to  the  DEARS;  and  the 
Contractor  Evaluation  Program  model,  developed  from  informadon  in  Section  n  C. 

The  business  case  analysis  focused  on  tiie  following  areas: 

•  Economic,  process  and  automated  data  information  systems  analyses  of  current 
past  performance  ^sterns. 

•  Analysis  of  recent  changes  to  the  FAR  and  proposed  changes  to  the  DEARS 
contractor  past  performance  relative  to  the  As-Is  model. 

•  A  comparison  between  the  Contractor  Evaluation  Program  and  the  FAR/DFARS 
model  programs. 

The  objective  of  the  economic  analyses  aspects  of  the  business  case  analysis  was  to 
provide  information  and  insight  that  would  help  determine  whether  the  use  of  past 
performance  information  in  the  procurement  process  makes  good  business  sense. 

We  first  reviewed  and  diagrammed  the  processes  for  the  systems  that  are  cunwatly  in 
use,  and  identified  the  principal  activities  that  are  involved  in  the  collection  and  use  of 
past  performance  information. 

In  examining  the  existing  systems,  the  major  cost  elements  were  found  to  be  related  to 
the  collection  and  validation  of  performance  information  for  possible  future  use  in 
source  selection  decisions.  The  major  steps  in  that  process  are: 

•  Evaluate  and  record  performance  of  contractors  on  active  contracts, 

•  Provide  opportunity  for  contractor  to  review  performance  information, 

•  Receive  response  from  contractors  in  the  form  of  comments,  rebuttal,  or 
additional  information  relative  to  their  performance, 

•  Review  any  response  from  contractors  and  resolve  contractor  evaluation,  and 
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•  File  all  infonnation  as  “Source  Selection  Information”  for  possible  future  use 
within  the  next  three-year  period. 

Attributing  specific  and  quantifiable  benefits  to  the  existing  systems  was  extremely 
difficult.  Factors  that  complicated  these  determinations  included  the  following: 

•  Some  of  the  information  systems  were  in  the  eariy  stages  of  implementation  and 
specific,  tangible  benefits  were  yet  to  be  demonstrated; 

•  The  value  of  specific  infonnation  was  difficult  to  isolate  because  of  the  multiple 
sources  and  types  of  infonnation  that  are  available  for  use  in  a  procurement 
process,  and  because  the  ultimate  award  decision  typically  involves  a  range 
information  from  many  sources 

•  The  existing  systems  were  tailored  to  specific  business  areas  and  used 
information  and  evaluation  factors  unique  to  business  areas  (which  did  not 
necessarily  have  relevance  outside  of  that  business  area). 

Whereas  these  systems  seem  to  be  operating  well  and  for  the  purpose  intended,  there 
was  concern  that  an  attempt  to  design  a  system  to  cover  all  active  contracts,  in  all 
product  and  service  codes,  will  prove  to  be  extremely  costly  with  very  limited  benefits 
that  can  be  supported  by  analysis. 

Because  of  the  design  features  of  the  Contractor  Evaluation  Program  model,  as 
previously  enumerated,  the  cost/benefit  ratio  appears  to  be  very  favorable.  This  is 
largely  because  tiie  specific  contractor  past  performance  information  issues  will  be 
addressed  and  resolved  at  the  business  area  level 

The  automated  data  information  system  portions  reviewed  and  documented  the 
information  technology  currently  used,  or  planned  for  use,  in  two  of  the  existing 
information  systems.  The  ultimate  resolution  of  issues  related  to  the  collection  and  use 
of  contractor  past  performance  information  will  clearly  benefit  from  the  application  and 
use  of  information  technology.  The  objective  of  this  task  area  was  to  explore  some  of 
the  possible  applications  for  this  technology  and  to  describe  the  potential  system 
development  options  that  appear  reasonable  and  feasible  at  this  point. 

The  comparison  of  existing  programs,  proposed  DFARS,  and  the  Contractor  Evaluation 
Program  was  the  final  aspect  of  the  business  case  analysis.  This  analysis  determined 
that  tiiere  were  criteria  that  could  be  used  to  evaluate  the  overall  performance  of 
contractors  and  that  these  criteria  could  be  reduced  to  conunon  data  elements  for  all 
types  of  products  and  services.  We  also  found  that  die  evaluation  criteria  must  be 
tailored  to  the  type  of  work  being  performed  in  order  for  the  information  to  be  useful  in 
making  contractor  selection  decisions  at  some  future  date.  We  concluded  that  it  is  not 
practical  to  strive  for  a  single  DOD-wide  past  performance  information  system  that 
prescribes  the  same  detailed  evaluation  criteria  and  common  data  elements  for  use  in 
evaluating  contractor  performance  in  all  acquisition  cases.  We  propose  collecting  that 
information  in  one  system. 

The  business  case  analysis  is  in  Section  n  D  of  the  report 
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D.  Conclusions 

One  of  the  principal  questions  that  the  study  addressed  was,  should  DOD  use 
information  on  the  past  performance  of  contractors  in  contractor  selection 
decisions?  Clearly,  the  answer  here  is  yes,  primarily  because: 

•  It  makes  good  business  sense  as  proven  by  overwhelming  industiy  acceptance 

•  It  is  being  used  successfully  in  DOD  now,  although  on  a  veiy  limited  scale 

•  It  can  be  tailored  to  fit  specific  circumstances,  although  proposed  DFARS  policy 
has  the  effect  of  limiting  tailoring. 

Other  related  and  more  detailed  questions  that  are  dealt  with  in  the  study  include,  what 
information  should  be  collected?,  what  type  of  ^proach  should  be  used?,  and  what 
direction  and  guidance  should  be  provided?  In  dealing  with  these  question,  we  believe 
that  the  following  general  principles  have  relevance  to  formulating  a  sensible  contractor 
past  performance  policy  within  the  DOD: 

•  Decentralize  -The  range  of  products  and  services,  and  the  variance  in  the  size, 
scope,  type,  and  complexity  of  contracts  makes  a  standard,  DOD-wide  system 
impractical.  Government  and  industry  experience  support  a  decentralized 
approach,  supplemented  with  general  guidelines,  decision  rules,  best  practices, 
and  information  technology  support 

•  Business  Areas  Focus  —The  implementation  of  past  performance  should  focus 
on  individual  business  areas  at  the  operating  level.  These  encompass  similar 
products  or  services  for  which  a  coherent  and  congruous  strategy  can  be 
developed  by  organizations  with  procurement  authority  and  technical 
responsibility. 

•  Total  Program  Context  --Past  performance  needs  to  be  viewed  in  the  context  of  a 
total  program  that  goes  beyond  the  collection  and  use  of  past  performance 
information,  and  covers: 

-  the  analysis  of  individual  business  areas  (both  internal  and  external  factors) 

-  development  of  a  sensible  strategy  for  contractor  past  performance  at  the 
business  area  level 

-  processes  designed  to  implement  the  strategy  for  business  areas  in  which 
the  organization  is  active 

•  Horizontal  Integration  --The  business  area  concept  starts  at  the  local  level,  where 
it  is  integrated  with  the  overall  acquisition  strategy  and  procurement  planning  for 
the  business  areas.  As  business  area  alliances  are  formed,  it  can  exerts  a 
horizontal  integration  effect  by  joining  similar  business  areas  across  the  services 
and  DLA.  The  implementing  direction  needs  to  emphasize  the  need  for  this 
integration  and  coordination. 

•  User-Driven  —The  users  of  past  performance  information  need  to  have  the 
principal  role  in  defining  what  information  to  collect,  when  to  collect  it,  and  how 
to  make  it  available  for  their  use  in  selecting  contractors.  And  the  users  should 
include  the  technical,  management,  and  procurement  officials  who  are  involved 
in,  and  responsible  for  making  contractor  selection  decisions.  In  this  regard,  the 
DFARS  policy  for  collection  is  separate  from  considerations  of  future  use. 
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in,  and  responsible  for  making  contractor  selection  decisions.  In  this  regard,  the 
DFARS  policy  for  collection  is  separate  from  considerations  of  future  use. 

•  Simple  —To  be  effective,  the  past  performance  approach  has  to  be  easy  to 
understand  and  explain,  without  being  simplistic,  or  it  runs  die  risk  of  being 
misunderstood,  ignored,  or  both. 

•  Sh^  Information  -Systems  and  processes  for  sharing  past  performance 
information  among  organizations  depend  on  all  of  the  above  and  should  be  dealt 
with  after  performing  the  steps  above. 


Section  in  of  the  report  expands  upon  these  conclusions. 
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SECTION  11.  A.  Government  Perspective 


1.  Establish  Past  Performance  Information  Definition 


Past  performance  information  is  relevant  information  regarding  a  contractor’s  actions 

under  previously  awarded  contracts.  It  includes; 

•  The  contractor’s  record  of  conforming  to  specifications  and  to  standards  of  good 
workmanship 

•  The  contractor’s  record  of  containing  and  forecasting  costs  on  any  previously 
performed  cost  reimbursable  contracts 

•  The  contractor’s  adherence  to  contract  schedules,  including  the  administrative 
aspects  of  performance 

•  The  contractor’s  history  for  reasonable  and  cooperative  behavior  and 
commitment  to  customer  satisfaction 

•  The  contractor’s  business-like  concern  for  the  interest  of  the  customer 

2.  Develop  Past  Performance  Information  Structure 


The  35  systems  and  processes,  listed  in  Appendix  A,  that  we  examined  contained  or  in 
some  way  dealt  with  past  performance  information  but  did  not  have  conqrarable 
information  system  structures.  To  add  to  tiie  challenge,  most  of  the  databases  contained 
the  information  in  multiple  categories,  e.g.  company,  contract  performance, 
administratrve.  We  found  it  useful  to  devise  a  scheme  to  classify  all  past  performance 
information  within  the  context  of  all  government  information.  This  information 
structure  is  shown  below; 
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Figure  11A.1:  Govenmiatt  Informafioii 

Once  the  chain  leading  to  past  peifonnance  information  for  use  in  contractor  selection 
was  established,  we  distinguished  between  past  performance  information  collected, 
validated,  and  filed  for  use  in  future  contractor  selections,  and  past  performance 
information  collected  and  used-possibly  validated-duiing  an  on-going  selection.  This 
distinction  is  depicted  in  the  diagram  below: 
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Figure  HA^:  Past  Performance  Information 
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3.  Identify  Existing  Systems  and  Processes 


Existing  Government  past  performance  information  systems  were  identified  using 
criteria  derived  from  OFPP  Policy  Letter  No.  92-5,  FAR  (FAC  90-26),  OFPP  “A  Guide 
to  Best  Practices  for  Past  Performance,”  and  interviews  with  Government  users,  system 
managers,  and  process  owners.  A  specific  distinction  was  made  between  past 
performance  information  systems  and  systems  that  just  had  past  performance 
information  elements. 

The  following  ciiteiia  was  used  to  specifically  identify  existing  past  performance 
information  systems: 

•  Information  is  collected,  validated,  filed,  and  disseminated  for  the  specific 
purpose  of  supporting  future  contractor  selection  decisions 

•  Opportunity  is  provided  for  review,  comment  and  rebuttal  of  the  information  by 
the  contractor 

•  Provisions  are  established  to  resolve  disputes  between  the  contractor  and  tiie 
govemmoit 

•  Information  is  subjected  to  the  same  controls  and  safeguards  as  other 
information  used  in  contractor  selection  decisions 

•  System  is  in  operation  and  currently  supporting  contractor  selection  decisions 

A  past  performance  information  system  is  an  ongoing  effort  to  collect  and  record  past 
performance  information  for  subsequent  use  in  determining  contractor  eligibility  and 
selection. 


Using  this  definition,  we  identified  three  types  of  past  performance  information  systems- 
-  which  include  six  distinct  systems  from  the  list  of  35  systems  we  reviewed.  The  three 
types  of  past  p^ormance  information  systems  are: 

•  Performance  appraisal  systems  containing  contractor  evaluations  prepared  by 
cognizant  government  offidals 

•  Performance  tracking  systems  which  draw  on  quality  and  delivery  data  from 
existing  databases 

•  Performance  certification  systems  which  establish  specific  criteria  which  are 
applied  for  purposes  of  identifying  high  levels  of  performance  exhibited  by 
certain  contractors 
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Using  this  structure,  the  following  past  performance  information  systems  were 

identified; 

•  Performance  appraisal  systems 

•  Contractor  Performance  Assessnwnt  Reporting  System  (CPARS)  developed 
and  used  within  the  Air  Force 

•  A&E  Contract  Administration  Support  System  (ACASS)  developed  and  used 
by  the  Army  Corps  of  Engineers 

•  Construction  Contract  Appraisal  Support  System  (CCASS),  also  developed 
and  used  by  die  Army  Corps  of  Engineers 

•  Performance  tracking  systems 

•  RedATellow/Green  (RYG)  system  developed  by  the  Navy  Supply  Systems 
Command  and  used  by  certain  procurement  oiganiz^ons  within  the  Navy 

•  Automated  Best  Value  Model  (ABVM)  developed  and  used  within  DLA 

•  Contractor  Profile  System  (CPS),  also  developed  by  DLA 

•  Performance  certification  systems  which  include  Blue  Ribbon  Programs  which 

have  been  implemented  by  some  procuremrait  organizations 
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Performance  ^praisal  systems,  such  as  CPARS  and  ACASS,  generally  cover  a  wide 
range  of  evaluation  factors— CPARS  addresses  14;  ACASS  rates  1 1 .  Performance 
tracking  systems  generally  focus  on  two  or  three  factors.  The  difference  in  the  number 
of  factors  is  generally  due  to  two  reasons:  tracking  systems  are  associated  with  higher 
volume,  relatively  small  dollar  acquisitions  ($25-$500K),  and  evaluations  may  be 
conducted  on  less  than  the  whole  contract  requirements.  The  performance  appraisal 
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systems  differ  from  tracking  systems  in  both  respects  in  that  the  evaluation  is  for  the 
whole  contract  and  it  is  used  in  low  volume,  relatively  high-dollar  value  contractor 
selections.  CPARs  uses  interim  reports  so  in  one  sense  is  also  based  on  less  than  the 
whole  contract,  but  the  final  CPARS  report  card  for  a  contractor  is  accomplished  for  the 
whole  contract  requirement  The  existing  systems  focus  on  a  specific  segment  of  the 
DOD  acquisition  program  in  terms  of  contractor  dollar  value  and  product/service. 

Performance  certification  systems  arc  generally  for  the  same  level  of  complexity  and 
dollar  value  as  tracking  systems.  They  build  upon  the  data  in  tracking  systems  but  go  a 
step  beyond  rating  and  ranking  contractors.  Performance  certification  systems  actually 
offer  pre-established  evaluation  standing  in  the  contractor  selection  process.  Certifying 
contractor  performance  requires  a  broader  level  of  information  than  is  obtained  in 
tracking  systems. 


4.  Coverage  of  DOD  Acquisition  Program  by  Existing  Government  Past 
Performance  Information  Systems 

By  1997  the  proposed  DEARS  policy  provides  that  past  performance  inform^on  must 
be  collected  for  all  active  contracts  ovct  $100,000.  The  challenge  to  cost-^ectively 
collect  that  ir^ormadon  is  die  major  issue  DOD  is  facing  in  implementing  past 
performance  policy.  That  is  why  it  is  important  to  imderstand  the  very  limited  coverage 
that  existing  past  performance  information  systems  provide  relative  to  the  total  DOD 
acquisition  program.  This  table  shows  the  limited  coverage  that  is  available  fiom 
current  systems  from  dollar  value  and  product/service  perspective. 


Contract  Dollar 
Values 

Research  & 
Development 

Services  and 
Construction 

Supplies  and 
Equipment 

>$5M 

CPARS’ 

>$100K 

ACASS 

CCASS 

<$100K 

RYG* 

ABVM* 

BRCP 

*  CPARS  is  also  used  on  major  R&D  programs 

^  The  contractor  performance  data  only  from  RYG  can  be  used  for  transactions  above  SlOOK 
^  AB  VM  can  be  used  for  higher  dollar  transactions  where  die  higher  dollar  value  is  due  to  greater  quantities 

An  approach  to  filling  the  extensive  g^s  in  coverage  in  research&development, 
services,  and  most  of  the  supplies  and  equipment  contracts  is  presented  in  Section  n  C- 
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5.  Systems  Assessments  and  Key  Characteristics  Evaluation 

The  results  of  system  assessments  and  key  characteristics  of  existing  government  past 
performance  information  systems  follow. 

System  Assessment— Red/Yellow/Green  (RYG)  This  is  a  Navy  system  designed  to  help 
reduce  the  risk  of  receiving  non  conforming  products  and  late  shipments.  RYG 
classifies  the  degree  of  risk  by  assigning  a  color  code  to  a  contractor's  historical  quality 
and  delivery  performance  in  individual  Federal  Siqiply  aassifications  (FSCs).  Red  is 
high  risk,  yellow  is  moderate  risk,  and  green  is  low  risk.  The  system  provides 
procedures  and  an  automated  system  for  incorporating  these  classifications  into 
contractor  selection  decisions. 

In  addition  to  the  color  indicators,  the  system  provides  price  adjustment  factors  that 
reflects  the  additional  cost  to  the  government  for  actions  needed  to  reduce  the  risk  of 
receiving  non  conforming  products  and  late  deliveries.  When  added  to  a  red  or  yellow 
offeror’s  price,  the  adjustment  factor  may  displace  the  low  offeror  in  favor  of  an  offeror 
widi  a  better  product  quality  and  delivery  history.  The  price  adjustment  feature  of  die 
system  genially  has  relevance  to  the  smaller  contracts  because  die  adjustment  factors 
represent  a  fixed  amount  and  this  amount  trads  to  lose  significance  as  the  contract  value 
approaches  $100,000. 

The  RYG  system  tracks  the  quahty  of  items  deUvered  under  specific  line  items  and  does 
so  by  relating  discrepancy  information  observed  and  reported  by  government  persoimel. 
This  information  is  used  to  calculate  a  rating  for  the  contractor's  specific  plant  location 
and  for  the  specific  FSC.  An  opportunity  is  provided  to  each  contractor  to  review  its 
ratings,  and  to  challenge  the  basis  upon  whidi  the  ratings  were  determined.  Differences 
are  resolved  between  the  government  and  the  contractor. 

The  ratings  and  the  related  price  adjustment  factors  are  then  available  for  use  by 
government  officials  as  a  factor  in  contractor  award  decision,  if  the  solicitation 
informed  the  contractors  that  this  past  performance  information  would  be  used  for 
making  the  award.  The  system  does  not  currently  cover  delivery  information,  although 
activity  is  underway  to  include  this  information.  The  system  does  not  have  the 
c^ability  to  track  the  in-service  quality  and  reliabili^  of  items  after  delivery  and 
acceptance,  although  this  type  of  information  can  be  retrieved  from  the  database  upon 
which  the  RYG  system  draws  its  data. 


The  RYG  system  has  been  in  operational  use  for-over.  five  years  and  implementation  and 
enhancements  are  continuing.  The  following  was  determined  during  the  course  of  the 
study; 

•  The  RYG  system  was  available  at  17  of  the  planned  41  sites,  but  some  are 
closing 
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•  Plans  for  expanded  use  at  more  sites  are  unclear  and  unscheduled 

•  Use  to  this  point  has  largely  been  by  advertising  the  system’s  value  versus 
directing  its  use 

The  RYG  system  includes  quality  data  and  is  currently  adding  delivery  performance, 
which  is  not  yet  operational.  It  combines  the  data  with  an  algorithm  that  produces  a 
color  indicator  (red,  yellow,  or  green)  and  a  Technical  Evaluation  Adjustment  (TEA),  a 
price  adjustmrat  added  to  the  bid  price  of  contractors  with  a  yellow  or  red  rating.  As  the 
dollar  value  of  the  contract  increases,  the  effect  of  the  TEA  in  an  award  decision 
decreases.  For  example,  contract  awards  over  $100K  do  not  use  the  TEA  feature.  Color 
code  ratings  can  provide  a  past  performance  indicator  for  any  contract  value.  When 
delivery  performance  is  incorporated  in  the  system,  two  sets  of  past  p^ormance 
indicators  will  be  provided  for  each  FSC  in  which  a  contractor  does  business  -  one  for 
quality  and  one  for  delivery.  RYG  Data  is  downloaded  monthly  to  the  using  acquisition 
offices.  Contractors  have  electronic  access  to  and  can  read  their  ratings.  The  RYG 
system  gives  indications  that  design  objectives  for  quality  related  issues  are  being  met, 
but  it  experlCTce  with  delivery  related  issues  is  too  limited  to  judge  its  ultimate  success 
throughout  the  Navy  widi  any  certainty. 

System  Assessment  -  AEVM:  This  DLA  system  was  also  designed  to  cover  spedfic 
equipment  and  supplies  with  FSCs  and  other  specifications,  hi  this  respect,  coverage  of 
the  system  is  similar  to  the  RYG  system.  It  includes  information  on  the  reported  quality 
and  on-time  delivery  of  specific  contract  line  items  and  uses  this  information  to 
calculate  a  score  for  each  contractor's  site  and  for  each  FSC.  An  opportunity  is  provided 
to  each  contractor  to  review  their  scores,  and  to  challenge  the  basis  upon  which  Aey 
were  determined.  The  scores  are  then  made  available  to  buyas  for  use  in  mating  award 
decisions.  The  ABVM  has  the  capability  to  include  the  results  of  random  testing  for 
items  accepted  and  maintained  in  stock. 

The  ABVM  is  a  module  in  the  DLA  Pre-Award  Contracting  System  (DPACS),  which  is 
the  migration  system  bang  used  in  the  development  of  the  Standard  Procurement 
System  (SPS).  Initial  implementation  of  ABVM  started  in  1995  and  is  currently 
underway  at  DLA  sites.  Consequently,  it  was  not  practicable  to  conduct  an  objective 
assessment  of  the  system  from  the  perspective  of  users.  A  user  survey  is  currently  being 
planned  by  DLA  to  assess  the  performance  of  the  system  and  to  solicit  ideas  for  system 
enhancements. 

The  ABVM  system  is  in  the  early  stages  of  operational  use.  For  exarrqile: 

•  Defense  General  Supply  Center  started  ABVM  operation  in  July  1995 

•  Training  programs  are  currently  underway 
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•  Other  DLA  Centers  will  have  installed  the  system  in  1996 

The  ABVM  system  includes  quality  and  delivery  performance  dfttn  which  are  combined 
with  an  algorithm  to  produce  a  score  for  each  contractor  in  each  FSC.  Past  performance 
scores  are  used  as  a  tool  in  making  a  comparative  assessment  of  price  and  performance 
risk.  ABVM  information  is  provided  to  buyers  through  DPACS.  Contractors  can  read 
their  ratings  through  an  electronic  bulletin  board.  ABVM  replaces  the  Quality  Vendor 
Program  (QVP)  as  the  principal  system  used  by  DLA.  QVP  is  a  performance 
certification  type  system  rather  than  a  performance  tracking  system.  DLA  shifted  to  the 
ABVM  because  QVP  was: 

•  Too  burdensome  to  administer 

•  Covered  only  a  small  portion  of  the  supplier  base 

•  Ended  up  with  two  ratings— certified  and  non-certified  rather  than  a  more 
comprehensive  ranking  of  the  suppliers 

QVP  is  still  authorized  for  use  for  specific  FSCs  or  selected  service  requirements  by 
individual  contracting  offices. 

ABVM  gives  indications  that  design  objectives  are  being  met,  but  it  is  too  soon  to  judge 
ultimate  success  throughout  the  DLA  with  any  certainty. 


System  Assessment  -  CPARS:  This  is  an  Air  Force  system  for  major  acquisition 
programs  above  $5  milhon.  that  was  designed  for  a  low  volume  of  transactions  and 
extensive  performance  measurement  categories.  It  is  a  manual  system  kept  in  files  at 
Air  Force  Product  Centers  which  provides  strong  support  to  the  source  selection  process 
by  communicating  contractor  strengths  and  weaknesses.  It  involves  a  periodic 
assessment  by  government  officials  responsible  for  the  overall  program;  uses  contractor 
data;  and  is  updated  every  12  months.  It  may  also  provide  out-of-cycle  reports.  It 
provides  relative  performance  feedback  to  contractors  across  all  measurements. 

The  measurement  factors  include: 

•  Product  /  system  performance,  including  system  engineering  and  software 
development 

•  Schedule 

•  Cost  performance 

•  Product  assurance 

•  Test  and  evaluation 

•  ILS  program 

•  Management  responsiveness 

•  Subcontract  management 


ArthirD  Little 


19 


Each  report  includes  a  description  of  the  program,  a  statement  describing  the 
contractor's  effort,  a  narrative  tiiat  addresses  the  performance  of  the  contractor  during 
the  period,  and  a  rating  for  each  of  the  evaluation  areas  listed  above  using  four  (X)lor 
codes-red,  yellow,  green  and  blue.  Performance  appraisals  are  provided  to  contractors 
for  their  review  and  comment,  and  then  reviewed  by  the  Government  evaluator  who 
may  adjust  the  initial  appraisal.  Conq)leted  reports  are  identified  as  "Source  Selection 
Information"  and  filed  in  libraries  maintained  at  AFMC  organizations  that  initiate  the 
assessment  report  Mormation  is  retrieved  for  use  in  source  selection  decisions  by 
contacting  the  cognizant  CPARS  focal  point. 

Some  initial  action  has  been  taken  to  automate  the  CPARS  process  utilizmg  Lotus  Notes 
as  well  as  to  extmd  coverage  to  small  systems,  services,  science  and  technology,  and 
operational  contracting. 

CPARS  very  consistently  performs  its  intended  purpose  as  reports  are  based  on  first¬ 
hand  data,  validated  by  contractors,  and  controlled  by  program  offices. 

System  Assessment  -  ACASS  &  CCASS:  ACASS  is  a  system  used  by  tiie  Army  Corps  of 
Engineers  which  cov^  architect  and  engineering  services  related  to  constructitm  (Code 
C  in  the  coding  structure  used  by  the  Federal  Procurement  Data  System  (FPDS)). 
Evaluations  are  pti^ared  by  professionals  who  review  and  accept  the  work.  Principal 
evaluation  areas  include: 

•  Thoroughness  of  site  investigations 

•  Quality  control  procedures  and  execution 

•  Accuracy  of  plans  and  specification 

•  Clarity  and  compl^ness  of  the  plans 

•  Overall  managemrat  and  adherence  to  schedule 

•  Compliance  with  cost  limitations 

•  Suitability  of  design  or  study  results 

•  Environmental  suitability  of  proposed  solution 

•  Cooperativeness  and  responsiveness  of  contractor 

•  Quality  of  briefings  and  presentations 


Evaluations  are  prepared  at  the  completion  of  contract  efforts.  Ratings  are  assigned  in 
three  categories  —  outstanding,  satisfactory,  and  unsatisfactory.  Contractors  have  the 
opportunity  to  review  and  challenge  the  evaluations.  Completed  assessments  are 
maintained  in  a  central  database  which  can  be  accessed  by  officials  who  are  involved  in 
contracting  for  A&E  s^vices. 

CCASS  is  also  a  system  used  by  the  Army  Corps  of  Engineers  which  covers  the 
construction  of  structures  and  facilities  (Code  Y  in  the  coding  structure  used  by  the 
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FPDS).  Evaluations  are  also  prepared  by  professionals  who  review  and  accept  the  work. 
Principal  evaluation  areas  include: 

•  Quality  of  work  (including  eleven  sub-factors) 

•  Timeliness  (including  seven  sub-factors) 

•  Effectiveness  of  management  (including  nine  sub-factors) 

•  Compliance  with  labor  standards  (including  three  sub-factors) 

•  Compliance  with  safety  standards  (including  three  sub-factors) 


Evaluatrons  are  prepared  at  the  conqiletion  of  contract  efforts.  Ratings  are  assigned  in 
five  categories-outstanding,  above  average,  satisfactory,  marginal  and  unsatisfactory. 
Contractors  have  the  opportunity  to  review  and  challenge  the  evaluations.  Completed 
assessments  are  maintained  in  a  central  database  which  can  be  accessed  by  officials  who 
are  involved  in  contracting  construction  wo±. 


These  systems  were  originally  designed  to  fadlitate  selection  of  “qualified”  A&E  and 
construction  contractors.  The  systems  were  lecmtiy  expanded  to  incorporate  past 
performance  information  into  contract  award  deosions.  Evaluations  are  performed  by 
government  professionals  responsible  for  reviewing  and  accepting  work: 

•  Administrative  Contracting  OfiScers 

•  Contracting  Officer’s  Representative 

•  Other  Receiving  Officials 

•  Resident  Engineers 


Reports  are  reviewed  with  contractors  and  entered  into  a  central  database  via  computer 
or  mail.  Access  to  the  data  is  provided  to  COE  elements;  contractors  do  not  have  read 
access  to  the  rating  information. 


System  Assessment-Contractor  Profile  System  (CPS):  The  Contractor  Profile  System,  a 
DLA  system  that  is  currently  available  for  use,  did  not  fully  meet  the  other  criteria  for 
past  performance  information  systems.  Woric  is  underway  by  DCMC  to  enhance 
MOCAS  data  extraction.  DCMC’s  Contractor  Information  Service  (CIS)  which  is 
currently  under  development  will  encompass  CPS  and  its  system  enhancements  which 
are  in  progress.  The  CIS  is  discussed  later  in  this  report  under  planned  past  performance 
information  systems. 


Key  Characteristics  Evaluation:  The  results  of  comparing  the  five  existing  DOD  Past 
Performance  Information  Systems  to  the  16  key  characteristics  provided  in  the 
Statement  of  Work  is  shown  below. 
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1  Existing  Past  Performance  Information  Systems 

1  System  Evaluation 

1  Factors 

RYG 

Navy 

ABVM 

DUV 

CPARS 
Air  Force 

ACASS 

CCASS 

CPS 

DLA 

Data  system  dMign  •  omtraitzed 
or  non-eentrallxsd 

Centralized 

Non-centralized 

CentraBzed 

Centiafized 

Centralized 

Kinds  of  data  used  - 
Govamnwnt,  private 

Quatity  and 
deUvery 

Quality  and 
delivery 

1 

Access  to  exist  - 
ing  databases. 

integrity  of  data  *  Manttty  of 
sources 

Drawn  from  ex¬ 
isting  databases 

Drawn  from  ex¬ 
isting  databases 

EvaLbyiespon  - 
sfoteoffidais 

EvaLbyrespon  - 
sMe  officials 

MOCAS.PASS. 

andDPACS 

Accuracy 

mSSSBESSm 

SSEtSSBSII 

Currency  -  fraquaney  of  update 

Monthly 

Monthly 

Based  on  source 
databases 

Remedial  process  by  contractors 

Yes 

Yes 

Yes 

Yes 

No  routine 
process  In  place 

Availability  of  information  for 
source  selection 

Via  computer 
terminal 

Via  computer 
terminal 

From  CPARS 
focal  point 

Vis  computer 
terminal 

Via  computer 
terminal 

Confidentitiity 

Yes 

Yes 

Yes 

Yes 

Yes 

Subcontractor  Involvement 

No 

No 

Yes 

No 

No 

Maintaining  identity  of  con 
tractors  that  are  acquired 

Yes 

Yes 

Yes 

No 

No 

Fairness 

Yes 

Yes 

Yes 

Yes 

Fairness 
ensured  by  user 

Due  process 

Yes 

Yee 

Yes 

Yos 

Due  process 
ensured  by  user 

Lack  of  past  psffemiancs 

Neutraf  rating 
used 

Average  score 
used 

NA 

Not  Included  in 
dstafasse 

imasnoNi  or  appncaowiy 

Primarily  below 

$100K 

$5Mand 

above 

Oiwr$2SK 

OLA  contracts  & 
sdmtabyOCMC 

CapabliKy  of  attrtbtilion 

Protected 

Protected 

Protected 

Mected 

No  performance 
evaluation  Info. 

Psnsity 

Info,  used  ki 
source  selection 

S.L-|I'M.|',H 

info,  used  in 
source  selection 

Figure  nA3:  Key  CSiaracteristics  of  Existing  Past  Performance  Information 
Systems 

Detailed  evaluations  for  each  system  were  briefed  to  DUSD(AR)  on  January  24, 1996 
and  included  in  that  deliverable. 


6.  Systems  Under  Development 

Each  of  the  DOD  components  have  initiatives  underway  to  implement  DEARS  that  are 
aimed  at  expanding  the  past  performance  information  available  for  use  in  contractor 
selection  dedsions. 

The  Air  Force  is  examining  an  automated  version  of  CPARS.  The  Navy  is  exploring  a 
Contractor  Evaluation  System.  The  Army  is  developing  a  Perfonnance  Information 
Management  System.  DLA  is  developing  the  Contractor  Information  Service. 

The  CPARS  approach,  used  for  major  acquisitions,  has  been  tailored  to  be  suitable  for 
small  systems,  services,  and  R&D.  The  tailoring  is  primarily  in  the  evaluation  factors 
that  are  addressed  in  each  case.  Some  initial  work  has  been  done  in  automating  the 
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records  that  would  facilitate  communication  with  contractors  and  the  filing  and  retrieval 
of  information. 

The  Contractor  Information  Service  design  goal  is  to  make  DCMC’s  knowledge  and 
experience  more  accessible  to  its  customers.  Goals  include: 

•  Near  term  -  enhance  MOCAS  data  extraction  capability  and  develop  past 
performance  input  screens 

•  Md  term  -  merge  three  existing  systems  (CPS,  PASS,  DSIS)  into  a  single 
information  system  over  the  next  two  to  three  years 

Information  will  be  organized  on  a  company-wide  basis  with  a  capability  to  “drill 
down”  to  divisions  and  plant  facilities.  Coverage  envisioned  includes: 

•  Principal  product  lines  and  unique  production  capabilities 

•  Company  organization  and  key  personnel 

•  Sales,  earning  and  financial  health 

•  Past  performance  history  -  trends,  data,  and  commentary 

•  Pridng  information  -  rates  and  factors 

•  Systems  and  processes  sti^  -  risk  assessments  and  corrective  actions 

•  Prior  reviews  -  GAO,  IG,  DCMC,  DCAA,  buying  activities 

•  Acquisition  strategy  “lessons  learned” 


7.  Experience  of  Other  Government  Agencies 


A  total  of  15  other  non-DOD  Federal  Agencies  were  reviewed  in  terms  of  their  approach 
to  past  performance  information  systems.  The  diversity  of  agencies  provided  extensive 
coverage  in  terms  of  the  nature  of  products  and  services  contracted  for  as  well  as 
missions  performed. 

Our  observations  are  based  on  contacts  with  officials  in  several  agencies  and  on  a 
review  of  documentation  on  past  performance  implementation.  Most  agencies  are 
implementing  past  performance  by  passing  the  OFPP  Guide  along  with  minimal 
guidance.  Some  exceptions  to  this  include: 

•  Energy-which  issued  a  10-page  Acquisition  Letter  to  accompany  the  OFPP 
Guide 

•  GSA-which  issued  an  Acquisition  Letter  in  March  1 993  (which  is  being 
updated)  and  Federal  Supply  Service  specific  guidance  in  a  separate  Acquisition 
Letter  issued  in  October  1995 
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Transportation—  guidance  issued  in  the  “Transportation  Acquisition  Manual” 


Generally,  the  evaluation  form  in  the  OFPP  Guide  is  provided  for  guidance  purposes. 
No  guidance  is  provided  on  how  to  tailor  the  evaluation  to  size,  content,  and  complexity 
of  the  contractual  requirements.  No  automated  databases  exist,  but  some  agencies  are 
planning  to  investigate.  Specific  concerns  include; 

•  Protests  and  ruling  by  appeals  boards  and  courts 

•  Numb^  of  open,  active  contracts  that  will  need  to  be  evaluated 

•  Workload  impact 

NASA  will  use  past  performance  in  source  selection,  but  will  not  evaluate  on-going 
contracts  except  as  required  for  award  fee  determinations.  NASA  implemented  a 
Contractor  Performance  Summary  (CPS)  in  January  1992  which  was  based  on  the  Air 
Force  CPARS.  CPS  features  included: 

•  An  evaluation  on  all  award  fee  contracts  above  $2SM 

•  Evaluation  of  non-award  fee  contracts  was  discretionary  (by  tire  Centers) 

The  CPS  system  was  abandoned  in  March  1994  because  the  value  added  to  the 
contracting  jnocess  could  not  justify  continuation  of  the  system.  In  response  to  OFPP  on 
the  recent  FAR  dianges,  NASA  will  continue  to  use  past  performance  as  a  standard 
evaluation  factor  in  source  selections  (NASA  has  been  doing  this  for  at  least  6  years). 
NASA  will  not  create  an  Agency-wide  system  to  require  performance  reports  on  active 
contracts. 

NASA  elected  not  to  require  performance  reports  because: 

•  Award  fee  evaluations  capture  approximately  80%  of  NASA’s  procurement 
dollars  (and  fliese  are  exempt  from  the  FAR) 

•  Implementing  an  Agency-wide  system  would  significantly  burden  the  workforce 
without  significant  benefits 


At  the  $100K  threshold,  80%  of  contracts  would  be  non-award  fee  contracts  and  would 
account  for  about  20%  of  the  procurement  dollars.  NASA  estimates  that  a  ten-fold 
increase  in  evaluations  would  be  needed  to  evaluate  the  non-award  fee  contracts  (from 
224  to  2404).  OFPP  requested  that  NASA  reconsider  their  decision.  This  seems 
unlikely. 

The  Federal  Supply  Service  in  GSA  has  issued  policy  on  use  of  past  performance 
information  and  on  a  system  for  routinely  recording  this  information.  Guidance  was 
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provided  by  the  FSS  Acquisition  Letter  FC-95-7  of  October  19, 1995,  “Use  of  Past 
Performance  as  an  Award  Evaluation  Factor  -  Routine  Stock  and  Special  Order 
Programs.”  The  guidelines  allow  a  contracting  officer  to  efficiently  use  the  quality  of  an 
offeror’s  past  performance  as  a  factor  in  a  contract  award  decision.  Past  performance  is 
to  be  considered  along  with  price  and  applies  to  negotiated  acquisitions  in  excess  of  $1 
million. 

The  process  provides  for: 

•  Supplier  Rating  Reports,  that  are  used  to  evaluate  contractor  performance  by  the 
cognizant  Office  of  Quality  and  Contracts. 

•  The  Administrative  Contracting  Officer  opinion  supported  by  a  summary  of  the 
Supplier  Rating  Report,  that  is  provided  to  a  Procuring  Contracting  Officer  upon 
request. 

•  The  Contracting  Officer’s  judgment  for  the  ultimate  award  decision. 


The  existence  of  well-developed  polity  was  very  limited.  Taken  as  a  whole,  these  15 
agencies’  systems,  with  the  exception  of  the  GSA  Federal  Supply  Service,  appear  to 
represent  less  structured  and  more  ad  hoc  past  performance  evaluation  programs. 
While  represrated  in  diis  report  at  a  summary  level,  most  of  diese  agencies’  programs 
^pear  to  represent  immature,  unstractured  tqipioaches  to  evaluation. 
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SECTION  II.  B.  Industry  Perspective 


This  section  presents  the  results  of  industry  benchmarking  and  our  review  of  industry 
supplier  evaluation  programs. 

1.  Benchmarking 


The  benchmaildng  phase  of  the  study  was  accomplished  by  conducting  on-site  visits 
and  by  reviewing  information  in  the  Arthur  D.  Little  Supply  Chain  Management  practice 
database. 


The  names  of  companies  we  visited  during  the  course  of  the  study  and  the  industries 
they  represent  are  listed  below.  Notice  that  while  manufacturing  is  heavily  represented, 
industries  dealing  with  electronics,  process  industries  and  con:q)anies  performing 
logistics-like  activities  were  included  in  our  research  to  provide  both  breadth  and  depth 
to  our  review. 

Companies  Included  in  the  Supplier  Evaluation  Database 
Company  Name  Industry  Type 

Alloi-Bradley  Process  Controls  Equipment  Manufacturing 

Baxter  Pharmaceutical  Manufacturing/Distrib. 

Black  and  Decker  Consumer  Goods  Manufacturing 

Boeing  Defimse  &  Space  Group  Aerospace/Defense  Manufacturing 

British  rail  Transportation 

Fisher  Scientific  Industrial  Distribution 


Ford  Motor  Co. 

McCormick  &  Co. 
McDonnell  Douglas 
Mobil  Corporation 
National  Semiconductor 
Rockwell  Defense  Electronics 
W.W.  Grainger 
U.S.  Postal  Service 


Automotive  h^ufacturing 
Consumer  Goods  Manufacturing 
Aerospace/Defense  Manufacturing 
Process  Manufacturing 
Electronics  Manufacturing 
Aerospace/Defense  Manufacturing 
Industrial  Distribution 
Transportation 
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Companies  researched  for  benchmarking  purposes  included  those  producing  consumer 
items  as  well  as  diose  in  the  defense  contracting  community.  In  addition,  companies 
manufacturing  component  parts  were  also  included  since  supplier  certification  is  often 
performed  down  to  the  part  level  in  a  system. 

A  key  finding  of  our  industry  research  is  that,  in  best-of-class  supplier  evaluation 
programs,  there  is  a  distinct  supplier  qiproval  process  keyed  to  associated  risks.  Also, 
the  supplier  approval  process  outcome  results  in  a  consolidation  of  suppliers,  which  is  a 
necessary  condition  before  a  business  relationship  can  take  place. 

2.  industry  Supplier  Evaluation  Programs 

TTie  information  in  this  section  is  organized  according  to  the  key  features  we  found  in 
industry  supplier  evaluation  programs.  The  nine  key  features,  which  were  identified  in 
con^ianies  th^  are  recogniised  as  “best  of  class”  among  supplier  evaluation  programs 
along  with  their  purpose,  scope,  and  selected  implementation  features  are  summarized  in 
the  following  table: 
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Evaluation  Program 
Component 


Selected  Implementation 
Features 


•  Align  supply  base  strategy  to 
coqsorate  strategy 

•  Manage  the  supply  base  to 
achieve  corporate  strategy 

•  Manage  the  st43ply  base  to 
create  leverage,  achieve 
least  total  cost  gain 
competitive  advantage 

•  Improve  continuously 

1 

Company^ftfWe 

AH  materials,  servloas,  equipment 

•  Size  of  supply  base, 
overall  and  within 
commodity  segments 

•  Commodities 
Management  Strategy 

•  Identification  of  key 
suppliers 

•  Long-term  partnership 
strategies 

•  Just-in-time 

•  ISO  9000,  Baldridge, 
other  certification 
requirements 

•  Process  control  focus 

•  Evaluate  supplier 
performance  on  an  ongoing 
basis 

«  Use  supplier  performance 
data  for  continuous 
improvement  total  cost 
reduction 

Typically  minimum  measurements 
Quality  of  product  materials/servlces 
provided 

Service  performance 

Deiivefy  performance 

Cost  performance 

Overall  commitment 

«  Measurement  can  apply 
to  alt  suppliers 

•  To  feedback  supplier 
performance  results  for  the 
purposes  of  Improving 
supplier  performance 

Strategic  and  Alliance  Suppliers 

«  Cross-functional  data 
sources 

«  Feedback  In  an  annual 
meeting 

•  Development  of 
corrective  action  plans 

•  Assure  that  the  suppliers* 
quality  systems  and 
processes  are  documented 

STKf  Inuse 

Comprehensive  assessment  of 
quality  systems,  often  based  on 

ISO  9000  or  other  systems 

•  QuaQty  Process  Self- 
Assessment 

Questionnaire 

«  Identify  parts  or  items  that 
consistently  meet  statistical 
process  control  requirements 
«  Identify  parts  or  items  based 
on  conformance  to  process 
that  will  not  be  subjected  to 
incoming  inspection 

All  suppfiers,  but  espedaSy  critical 
suiters 

•  100%  conformance  to 
requirements 

•  Determine  the  total  cost  of 
doing  business  in  a  supplier 
relationship 

AH  suppliers  but  especially  critical 
suffers 

•  Published  guidelines 
and  standards 

•  Formal  new  supplier 
education  program 

«  Communicate  supplier 
performance  standards  and 
requirements 

•  Educate  suppliers  on  the 
supplier  Improvement 
process 

All  suppliers,  but  especially  critical 
suppliers 

•  Published  guidelines 
and  standards 

•  Formal  new  supplier 
education  program 

•  identify  the  si^pliers  that  are 
approved  to  buy  from  before 
orders  are  placed 

Company-wide 

Suppliers  quality  system 

Sup^ier's  conformance  to  regulatory 
requirements 

Supplier’s  general  business  standing 

•  Depth  of  evaluation 
varies  with  risk 

•  Honor  the  best  performing 
suppliers 

•  Recognize  outstanding 
contributions  by  a  supplier 
employee 

Strategic  and  critical  sippiiers 

•  Plaques,  certificates  of 
appreciation,  thank  you 
notes 

Fi^^ire  IIB.l :  Key  Features:  Supplier  Evaluation  Program 
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Supply  Based  Management  Process^upply-Based  Strategy 

In  terms  of  “best-of-class”  benchmarking  findings,  the  following  three  principles, 

derived  from  the  first  key  feature  listed  above,  stand  out: 

•  World  class  supply  chain  orientafirm 

•  Supply  base  improvement  strategy 

•  Explicit  supply  base  management  process 


The  emphasis  on  a  worid-class  orientation  moves  organizations  fi:om  a  prescriptive 
“m^t  the  spec”  environment  to  a  fully  collaborative  internal  and  external  team 
environment  which  emphasizes  process  rather  than  specifications..The  emphasis  on 
developing  an  explicit  supply  base  improvement  strategy  and  management  process 
raises  the  level  of  supplier  performance,  reduces  supply  chain  costs,  and  moves  staff  and 
suppliere  into  new  roles  which  change  over  time  fi-om  a  largely  reactive  to  a  proactive 
orientation  that  reinforces  continued  in:q)rovements.  Conspicuous  in  this  new 
perspective  is  a  systems  approach  to  delivery  of  world-class  products  and  services. 

Supplier  Performance  Measurement 

Ongoing  supplier  performance  measurement  is  a  coitral  feature  of  supplier  evaluation 
progr^.  The  scope  of  application  ranges  from  all  suppliers  to  critical  suppliers 
Supplier  performance  measurement  is  generally  p«formed  for  a  small  numbn  of  critical 
data  elements,  such  as  quality,  service,  delivery,  and  cost  Each  business  unit  defines 
what  constitutes  product  quahty,  SCTvice,  and  delivery  performance,  as  well  as  the 
appropriate  measures  for  each  of  these. 

Prior  to  beginning  supplier  paformance  measurements,  the  customer’s  performance 
standards  Md  requirements  are  communicated  to  suppliers.  Standards  include  how  a 
supplier  will  be  rated  and  how  tarings  will  be  used  and  communicated. 

Detailed  profiles  of  ratings  are  generally  available  on-line.  A  supplier’s  data  is  never 

shared  with  another  supplier.  Examples  of  these  profiles  are  shown  in  the  two  foUowing 
figures: 
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Figure  IIB JS :  Supplier  Performance  Measurement  Profile  OVlanufacturer) 


In  Figure  IIB.2,  a  supplier’s  paformance  index  (SPI)  and  an  average  for  the  commodity 
group  are  calculated.  Each  commodity  group  has  a  cutoff  or  "redline.”  Suppliers  with 
an  SPI  above  the  ledline  are  not  eligible  for  awards. 

A  sample  supplier  performance  report  used  by  a  national  distributor,  shown  in  Figure 
IIB.3,  has  four  categories.  Three  categories  pertain  exclusively  to  the  supplier’s 
performance  in  terms  of  quality,  cost  and  delivery  (timeliness  or  schedule).  The  last 
category,  sales,  pertains  to  the  buying  company’s  performance  associated  with  a  product 
line  involving  a  particular  supplier. 
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Figure  nB3:  Supplier  Performance  Measurement  Profile  (Distribution) 

Measures  in  the  “criteria”  column  of  tfie  report  are  accompanied  by  Hat}^  in  a  format 
which  has  the  capability  to  indicate  existing  trends.  In  addition  to  using  measures 

appUcable  to  operating  units,  the  report  also  includes  data  in  doUars-suitable  for  use  bv 
upper  management. 

Suppliers  Performance  Measurement  Feedback 

Another  key  feature  of  a  supplier  evaluation  program  is  focused  on  feedback  pixrcesses 
and  improvements  in  communication.  Feedback  to  suppliers  is  a  very  important 
mgredient  in  an  effective  supplier  evaluation  program.  This  provides  needed 
information  on  quality  to  suppliers  for  their  own  improvement  processes.  Best-in-class 
companies  provide  feedback  to  their  suppliers  on  their  performance  results  for  the 
purpose  of  improving  future  performance.  An  effective  suppher  evaluation  program  will 
have  to  contend  with  both  the  nature  of  specific  feedback  as  well  as  the  fiequency 
Many  organizations  utilize  a  formal  “report  card”  process  to  provide  suppliers  with 
feedback  in  a  structured  fashion.  Many  companies  meet  with  their  suppliers  at  least 
once  a  year  to  inform  them  of  their  evaluation  results,  identify  areas  of  improvement, 
and  m  more  advanced  situations,  develop  an  action  plan  for  improvement.  Companies 
also  notify  their  suppliers  more  frequently  by  on-Une  services,  telephone,  or  letter  about 
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their  performance.  This  feedback  is  critical  since  it  gives  both  parties  the  opportunity  to 
in^)rove  the  product,  reduce  costs,  and  improve  service 

Supplier  Quality  System  Assessment 

The  foundation  for  a  supplier  evaluation  program  lies  in  an  active,  thorough,  on-site 
evaluation  of  a  supplier’s  ^proach  to  the  installation  and  use  of  an  effective  quality 
system.  Supplier  quality  systems  assessments  are  often  based  on  rigorous  standards  such 
as  the  ISO  9000  series  of  standards.  A  key  feature  of  the  ISO  series  is  registration  of  a 
company  or  production  element  with  a  third  party  organization  which  monitors 
compliance  to  the  registered  standard.  Purchasers  of  products  and  services  from  ISO 
registered  companies  are  assured  that  the  registered  conqrany  has  a  documented  quality 
system  in  place.  Some  approaches  to  assessment  are  developed  in-house  using  ISO 
9000  (or  other  applicable  standards  for  the  industry)  or  the  ^^colm  Baldridge  National 
Quality  Award  criteria.  The  most  objective  approaches  at  this  time  rely  on  third-party 
certification  including  on-site  evaluation,  subsequent  registration,  and  periodic  re- 
evaluation. 

Part-level  Certification 

Part-level  certification  requires  accurate  historical  data  on  supplier  past  performance. 

An  inqwrtant  outcome,  often  not  explicitly  stated,  is  tire  change  in  the  relationship 
which  occurs  as  a  result  of  becoming  a  certified  supplin.  Generally,  companies 
requiring  supplier  certification  often  experience  a  decrease  in  the  number  of  qualified 
suppliers.  The  remaining  suppliers,  have  an  opportunity  to  develop  a  more  stable 
business  relationship. 

Suppli^  certification  tends  to  bring  increased  benefits  for  both  the  certified  supplier  and 
the  customer.  For  the  supplier,  it  can  mean  additional  business,  single  or  lead  source 
within  a  commodity  area.  For  the  customer,  it  can  mean  significant  cost  savings  by 
using  parts  received  from  certified  suppliers  because  certification  can  eliminate  costly 
incoming  inspection  and  associated  costs.  The  best-in-class  supplier  evaluation 
programs  usually  certify  to  the  item/part  or  family  of  parts  level.  Most  companies  have 
the  goal  of  certifying  all  of  their  key  parts  and  products.  However,  they  typically  start 
with  a  manageable  number  of  critical  parts  and  then  expand  the  program  to  include  all  of 
the  critical  items  as  well  as  those  that  have  the  potential  to  reduce  operating  costs.  Some 
companies  interviewed  during  the  course  of  the  study  had  certified  virtually  all  of  their 
products  or  were  on  their  way  to  certifying  all  critical  parts. 
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Total  Cost  Assessment 

An  emerging  trend  in  the  suppliw  evaluation  arena  is  the  use  of  a  “total  cost 
assessment”  approach  which  attempts  to  capture  all  of  the  acquisition  and  consumption 
costs  associated  with  doing  business  with  a  particular  supplier.  Acquisition  costs  are  the 
costs  of  a  supplier’s  activities  to  process  and  deliver  an  order  and  supplier’s  material  and 
profit.  Consumption  costs  are  the  costs  of  the  customer  activities— labor  and  overfiead— 
to  process  a  supplier’s  shipment  through  the  customer’s  system.  Effective  total  costs 
assessment  processes  usually  rely  on  activity-based  costing  principles.  Activity  based 
costing  techniques  are  used  to  acquire  the  best  value  by  estimating  the  total  costs  of 
doing  business  with  different  suppUers.  The  “tree”  lowest  bidder  is  sought-and  bids 
account  for  ^  costs  including  quality,  cost,  and  delivery.  Customers  identify  historical 
non-productive  costs  resulting  from  supplier  non-compliance  with  customer’s  mode  of 
operations.  Some  supplier  non-productive  events  that  are  “charged”  to  the  supplier  are: 


•  Source  r^'ection 

•  Early  delivery 

•  Inspection  resubmrttal 

•  Overshipment 

•  Return  to  supplier 

•  Late  receipt 

•  Material  review 

•  Shop  floor  rejection  (latent 

defect) 

•  Corrective  action  request  tetter 

•  Supplier  stop  notice 

One  important  use  of  a  total  cost  assessment  is  the  adjustment  of  bid  prices  from 
suppliers  using  a  SuppKer  Performance  Index  (SPI).  The  index  is  developed  from  a 
ratio  that  estimates  the  true  cost  of  supplier  bids.  An  example  application  of  the 
Supplier  Performance  hidex  concept  is  illustrated  in  Figure  IIB.4. 
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spn  = 


Material  Cost  +  Nonproductive  Cost 


Material  Cost 


'  A  lot  normalization  factor  (Q  factor)  is  used  to  eliminate  any  lot  value  bias  (Not 
shown) 

^  For  this  company,  suppliers  with  insufficient  data  for  a  valid  SPI  are  weighted  at 
the  commodity  group  average. 

Figure  IIB.4:  Total  Cost  Assessment  Ratio 
Supplier  Evaluation  and  Development 

There  are  two  primary  purposes  to  supplier  evaluation  and  development  initiatives; 

•  Conununicate  supplier  performance  standards  and  requirements;  and, 

•  Educate  suppliers  on  the  supplier  improvement  process. 

The  scope  generally  covers  all  suppliers,  but  especially  critical  suppliers.  Companies 
typically  communicate  their  guidelines  and  standards  through  published  documents  and 
formal  supplier  education  programs.  This  is  a  highly  proactive  process  in  which 
companies  view  their  suppliers  “as  their  customers.” 

Supplier  Approval 

A  robust  supplier  approval  process  incorporates  multiple  data  sources,  focuses  on 
quality,  is  documented,  and  is  shared  with  suppliers.  Supplier  information  gathered 
during  the  evaluation  may  include  general  business  standing,  service  levels, 
distribution/logistic  capabilities,  supplier  specifications/product  brochures,  company 
networking,  and  existing  like-product  data. 


An  example  of  the  scope  and  depth  of  supplier  approval  programs  is  presented  in  Figure 
IIB.5: 
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Note:  Example  from  manufactuiei/distdbutor  operating  under  Current  Good  Manufacturing  Practices  (CGMP). 

Figure  nB^:  Supplier  Approval  Elements 


Approval  is  formally  documented  to  cover  approved  locations;  any  required  reports  or 
data;  a  list  of  processes  approved;  additional  relevant  quality  information;  and  sign-off 
by  business  area  teams. 


Supplier  Recognition  Programs 

Many  commercial  firms  acknowledge  supplier  performance  with  a  recognition 
program.  Recognition  programs  vary,  but  an  important  outcome  is  the  strengthening  of 
customer-supplier  relationships.  Many  companies  present  their  best  performing 
suppliers  with  an  award,  while  others  less  formally  send  thank  you  letters.  Most 
suppliers  strive  for  such  recognition-it  brings  publicity  as  well  as  more  business  from 
the  customer  giving  the  award. 
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Section  li.  C,  Contractor  Evaluation  Program 


1.  Introduction 


To  develop  a  “To-be”  vision  for  past  performance  policy  implementation,  we 
proceeded  through  several  phases.  The  first  phase  involved  analysis  of  government  and 
industry  information  including  document  reviews,  research  results,  system  assessments, 
benchmarking,  and  interviews.  The  analysis  led  to  the  development  of  concepts  within 
a  tentative  model.  The  second  phase  involved  designing  a  workshop  approach  with 
ARSSG  representatives  and  preparing  materials  to  facilitate  the  workshops.  The  third 
phase  involved  scheduling/conducting  the  actual  workshops,  and  collecting  perspectives 
and  insights  concerning  “To-be”  concepts  and  issues  from  the  workshop  participants. 
The  final  phase  involved  integration  of  information,  perspectives  and  concepts  into  the 
actual  “To-be”  model— the  Contractor  Evaluation  Program 

Monnation  essential  to  developing  the  Contractor  Evaluation  Program  model  was 
collected  over  the  course  of  the  study  and  described  in  the  preceding  sections  of  this 
repo^  The  worlcshops,  with  ARSSG  representatives,  added  value  to  the  process  by 
providing  a  broader,  functional  partutipation  than  had  been  present  in  previous  past 
performance  forums.  The  ARSSG  workshops  included  representatives  from  Major 
Programs  (API),  Logistics,  Economic  Security,  Systems  Engineering,  Quality,  Inspector 
General,  Procurement,  General  Counsel,  Defense  Contract  Management  Command 
(DCMC),and  Defense  Contract  Audit  Agency  in  the  workshops.  We  were  eventually 
able  to  test  and  explore  the  implications  of  the  vision  of  a  “To-be”  model  on  surrogates 
for  the  DOD  acquisition  management  and  user  communities.  The  workshop  format  also 
provided  a  fomm  for  identification  of  milestones  supporting  the  vision  and  discussion  of 
actions  to  be  taken. 

The  workshop  approach  provided  an  opportunity  to  demonstrate  the  status  of  current 
activity  in  both  the  government  and  industry  arenas  and  led  to  the  identification  of  many 
important  factors  for  the  Contractor  Evaluation  Program  model,  including  the  following: 

•  A  key  objective  for  industry  in  adopting  supplier  evaluation  programs  is  to 
increase  competitiveness.  Characteristics  associated  with  increased 
competitiveness  include: 

•  Reduced  costs 

•  Reduced  cycle  and  response  times 

•  Improved  operational  efficiencies 

•  Increased  customer  satisfaction  and  loyalty 
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•  Reduced  inventoiy  (improved  inventory  turnover) 

•  Increased  revenues 

•  Three  key  factors  are  inherent  in  the  buyer-supplier  relationship.  These  are 

collaboration,  competency,  and  continuous  improvement.  Characteristics 
associated  with  these  include: 

•  Willingness  to  invest  resources 

•  Quality  products 

•  Service 

•  Responsiveness 

•  Technology 

•  Corporate  culture 

•  Industry  uses  supplier  evaluation  programs  to  meet  specific  objectives. 
Components  of  industry  supplier  evaluation  programs  are: 

•  Business  area  and  management  strategy 

•  Qualification  Assessment  (single  quality  system) 

•  Performance  measurement 

•  Performance  feedback 

•  Item  certification 

•  Total  cost  assessment 

•  Supplier  recognition 

•  Supplier  evaluation  and  development 

•  Supplier  approval 

•  The  uses  for  past  performance  in  DOD  are: 

•  Evaluate  risk  of  performance.  Provide  information  which  can  be  used  in 
making  trade-off  decisions  for  what  is  the  best  value  in  the  source 
selection  process.  This  information  can  be  used  in  the  award  of  the  initial 
contract,  exercise  of  options,  and  the  issuance  of  task  and  delivery  orders. 

•  Develop  acquisition  strategy.  Help  in  the  decision  as  to  contract  type 
and  source  selection  factors,  e.g.,  the  mid-1980s  overuse  of  FP 
developnKMt  contracts  led  to  many  cost  overruns. 

•  Manage  contractor  performance.  Provide  information  to  identify 
variances  fi-om  established  tolerances  in  the  existing  program. 

•  Improve  contractor  performance.  Provide  feedback  to  the  contractor 

about  performance  to  allow  the  contractor  an  opportunity  to  improve  its 
performance. 
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•  Allocate  oversight  and  review  resources .  Identify  those  contracts  or 
aspects  of  contracts  in  which  experience  dictates  there  have  been 
problems,  and  employing  oversight  resources  in  those  areas. 

•  Streamline  the  source  selection  process.  Reduce  reliance  on  voluminous 
contractor  technical  or  management  proposals  that  may  not  lead  to 
expected  contract  performance. 


2.  Contractor  Evaluation  Program 

The  breadth,  dq>th,  and  complexity  of  requirements  is  a  major  challenge  to  those 
involved  in  DOD  acquisition  programs  and  to  those  proposing  solutions  to  issues  (such 
as  the  past  performance  policy  implementation  issues  being  considered  in  this  study). 
The  Contractor  Evaluation  Program  we  designed  is  aimed  at  simplifying  the  past 
performance  implementation  effort  facing  the  DOD  as  well  as  to  improve  the 
effectiveness  of  this  effort  The  program  is  conducted  by  cross-functional  Business 
Area  Teams  tiiat  start  locally  and,  as  business  area  alliances  are  formed,  may  extend 
across  the  DOD  components,  as  appropriate. 


Overview 

Implementation  of  the  recent  policy  on  contractor  past  performance  requires  a 
recognition  of  the  business  environment  and  existing  acquisition  systems,  to  include: 

•  The  total  size  of  the  defense  business 

•  The  wide  range  of  products  and  services  for  which  contracts  are  issued 

•  The  large  number  of  procurement  organizations  that  issue  contracts 

•  The  existing  procurement  process 

In  the  aggregate  these  factors  define  a  conglomerate  that  is  engaged  in  an  extensive 
number  of  business  areas.  In  recognition  of  these  factors,  and  with  the  overall  goal  of 
reducing  the  cost  of  doing  business,  the  Contractor  Evaluation  Program  is  designed  to: 

•  Develop  a  Business  Area  Plan,  including  defining  common  business  areas 

•  Develop  a  Business  Area  Strategy  that  makes  sense  for  the  particular  business 
area 

•  Develop  a  Business  Area  Evaluation  Process  that  implements  the  business  area 
strategy. 
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Define  a  process  for  eadP^ 
business  area  to  evaluate 
peflofmance  and  to  use] 
this  information  for  source 
seiection  purposes 


The  Contractor  Evaluation  Program  is  designed  to  ensure  the  business  area  is  getting  the 
information  to  select  world-class  suppliers  with  a  best-value  outcome.  The  scope 
includes  the  products  and  services  acquired  by  all  the  services  and  agencies. 

Business  Area  Plan 

There  arc  four  aspects  to  the  business  area  plan 


Define  your  business  area  _ 

Develop  a  business  area  resource  center 

Conduct  business  area  analyses _ 

Form  business  area  alliances. 


Define  Your  Business  Area:  Defining  your  business  area  is  the  step  designed  to  take  an 
organization  from  a  vertical  hierarchical  focus  to  a  horizontal  view  of  acquisition 
programs,  products,  and  services  in  its  local,  inter-Command,  inter-Service/Agency,  and 
inter-Service/ Agency  acquisition  environment.  An  example  to  illustrate  the  process  for 
defining  your  business  area  follows. 
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The  wide  range  of  products  and  services  purchased  in  DOD  is  the  basis  for  starting  this 
exan^le  of  defining  a  business  area. 
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Figure  nC.l:  DOD  Products  and  Services 
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These  are  some  of  the  major  acquisition  organizations  within  the  DOD  components 
Aniiy  I  I  Navy  I  |  Air  Force  I  I  DLA  I 


Amiamont,  MunWons  &  Chem.  Com. 
Armament  R  &  D  Center 
Chemical  Biological  Defense  Com. 
Aviation  and  Troop  Commarxl 
Communications  &  Bectnonlcs  Com. 
Missile  Command 
Tank-Automotive  Command 
Troop  Support  Command 
Bafllstfc  Research  Laboratory 
Research  Laboratory 
Belvolr  Research  Dev.  &  Eng.  Ctr. 
Defense  Supply  Service  Wash. 
Military  Traffic  Management  Com. 
Mertical  ROA&L  Cornmand 
Health  Services  Commaixls  ' 

Brooke  Medcal  Center  ’ 

Central  Contracting  Offioe  ’ 

Elsenhower  Medical  Center  * 

Btzsimons  Medkal  Center 
Beaumont  Medical  Center  < 

Madigan  Medical  Center 
Tripler  Medical  Center  ' 

Walter  Reed  Medical  Center 
U.S.  Arnty  Depots 
Anniston  Depot  * 

Corpus  Christ!  Depot  * 

Latterkenney  Depot 
Red  River  depot 
Tobyhanna  Depot 
Tooele  Depot 

Corps  of  Engineers  i 

Information  Systems  Command  4 

Info.  Sys.  Selection  &  Acquisition 

*  Navy  activities  that  use.  or 
have  used  the  RYG  system 


i.  Headquarters  U.S.  Marine  Coip 
MHitary  Sealift  Command 
Office.  Chief  Naval  Research 
Strategic  Systems  Program  Office 
i.  Naval  Air  Systems  Command 
Speoe  &  Naval  Warfare  Sys.  Com. 
Naval  Fadlties  Engineering  Com. 
Naval  Sea  Systems  Command 
Regional  Contracting  Ctr..  Wash. 
Regional  Contracting  Ctr..  PM. 
Regional  ConL  Ctr..  Long  Beach 
a  Na^  Aviation  Sipply  Offioe 
O  Navy  Ship  Parts  Control  Center 
4t  Maie  Mand  Naval  Shipyard 
O  Pearl  HartXK  Naval  Shipyard 
«  Portvnouth  Naval  Shlp^ 

«  PhladalpMa  Naval  Sh^jyard 
4t  Norfbic  Naval  Shipyard 
Submarlna  Support  FadKy.  Groton 
*  Air  Warfaro  Canter.  Lakehurst 
Air  Warfare  Cantor.  Wuminstor 
4c  Air  Warfare  Center,  fndtenapofls 
Air  Warfare  Cerder,  Pax.  River 
Air  Warfare  Cantor,  China  Lake 
«  Air  Warfare  Center,  Orlando 
4c  Surface  Warfare  Center,  Crane 
Surface  Warfare  Ctr..  Indten  Head 
Surface  Warfaro  Ctr.,  Panama  City 
Surface  Warfare  Cantor.  Oahlgien 
Surface  WSarfue  Center.  CardacDcfc 
«  Surface  Warfiue  Center.  touMIs 
4c  Undstaea  WaifMe  Or.,  Kqrpott 
Undereaa  Wartere  Ctr^  Newport,  Rl 
Eachanga  8aiv.  Com..  Wl  Beach 
Ooaanogfaphfo  Oflfoa,  Stennia.  MS 
Command.  Cortrol  A  Ooaan  Sura.  Ctr. 
Headquartera.  Naval  OteUcI  Wash. 

«  RestA  InduteM  Sup.  Ctro.  (9  atiss) 
Conatuction  Otganlzationa(6  altet] 


I  Air  Force  | 

Oklahoma  City  Air  Logistics  Center 
Ogden  Air  Logistics  Center 
Sacramento  Air  Logistics  Center 
San  Antonio  Air  Lo^^stics  Center 
Warner  Robins  AJr  Logistics  Center 
Aerospace  Guidartoe  and  Metrology 
Center,  OH 

Space  and  Mssie  Systems  Center 
Electronic  Systems  Center 
Aeronautical  Systems  Center  (ASC) 
Directorate  of  R&O,  ASC 
Air  Force  Dev.  Test  Center,  Egfin 
Air  Education  A  Training  Com.,  TX 
Air  MRitary  Command.  Scott  ATO 
Air  Combat  Command,  Langley.  VA 
10th  Air  Base  Wing.  USAF  Academy 
Air  Intelligenoe  Agency 
Air  Force  Space  Com..  Petetaon  AFB 
Air  Force  Reserve,  Robbins  AFB 
Spece  and  Missile  System  Center,  U 
HQ  Space  Command,  Peterson  AFB 
Human  Systems  Center.  Brooks  AFB 
Armstrong  Laboratory,  Brooks  AFB 
Phnifos  Laboratory.  Nrttand  AFB 
Wright  Laboratory.  WP  AFB 
Freight  Test  Center,  Edwards  AFB 
Romo  Air  Oevelopmertt  Center 
Arnold  Eng.  Dev.  Ctr..  Arnold  AFB 
CIvI  Engirieeting  Center.  Tyndaf  AFB 
Office.  Scjerrtffic  Reseandi.  BolHng  AFB 


DefeoM  I 
Contracting  I 
Mgmt  Command! 


DLA 

(Contracting) 

--General  Supply 
Center,  Ri^mond 

-Construction  Supply 
Center.  Colurrbus 

-  Electronics  Supply 
Center,  Dayton 

“  Fuel  SuppV  Center, 
Alexant^  VA 

-  Industrial  Supply 
Center.  Phlla. 

-  Personnel  Support 
Center.  Phila. 

-Subslstenca  Region 
Pacific.  Alameda 

-  ADP/T  Contracting 
Office.  Alenndfia 

-  National  Stodqple 
Center.  Art.  VA 


Figure  nC^:  DOD  Acquisition  Organizations 


In  the  business  area  definition  process,  these  acquisition  organizations  are  initially 
categorized  by  System,  Central,  Base/Post/Camp,  and  Science  and  Technology. 
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This  is  a  business  area  depicted  at  a  commodity  level: 


Types  of  Procurement  Organizations  and  Products/Servlces 
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Develop  a  Business  Area  Resource  Center.  Extensive  infonnadon  is  required  to  keep 
the  business  area  teams  supplied  with  internal,  industiy,  and  contractor  information  they 
will  need  to  conduct  their  analyses.  Each  business  area  should  have  an  on-line  or  other 
of  resource  center  to  keep  its  implementation  up-to-date  and  to  support  its  business  area 
analysis. 


Examples  of  the  data  elements  that  may  be  needed  for  industry  analysis  are: 
information  on  competitors,  market  size/growth,  market  forecasts,  profitability,  cost 
structure,  and  technology.  Examples  of  contractor  analysis  data  elements:  market  share, 
balance  sheet,  facilities,  profitability,  and  size/growth  information. 


The  sources  for  such  data  are  internal  documents  and  external  references  such  as  Duns 
Business  Rankings,  S«&P’s  Industry  Surveys,  Ward’s  Directories,  U.S.  Industrial 
Outlook,  Producers  Prices,  and  Prices  Index. 


Conduct  Business  Area  Analyses:  The  objective  of  the  business  area  analysis  is  to 
develop  an  understanding  of  the  internal  and  external  aspects  of  the  business  area,  and 
evaluate  and  improve  the  performance  of  contractors.  The  Business  Area  Analyses  can 
help  evaluate: 

•  What  is  the  past  experience  and  future  requirements  of  the  government  in  this 
area? 
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•  What  are  the  relevant  characteristics  of  the  industiy  in  terms  of  size,  growth,  and 
competitive  forces? 

•  What  is  the  current  position  of  the  key  contractors  in  the  industry? 

Some  of  the  factors  that  are  typically  involved  in  this  analyses  include: 

Internal  analysis 

•  Current  contractor/supplier  base 

•  Government’s  past  experience 

•  Expenditures  over  time 

•  Internal  acquisition  costs 

•  Projected  requirements 

External  analysis  (industry) 

•  Market  size  and  growth 

•  Capacity  and  utilization 

•  Market  share  of  piindpal  contractors 

•  Industiy  profitability 

•  Cost  stmcture  and  drivers 

External  analysis  (contractor) 

•  Customer  base 

•  Position  in  the  industiy 

•  Commitment  to  industiy 

•  Quality  and  service  performance 


Form  Business  Area  Alliances:  Business  areas  may  start  as  a  local  construct,  but 
regarding  the  business  area  as  only  a  local  construct  limits  the  synergy  that  exists  from 
the  horizontal  integration  of  DOD-wide  resources.  At  its  fullest  expression,  a  business 
area  will  bring  together  the  expertise  in  the  DOD  components,  accelerate  the  elimination 
of  waste  and  inefficiency,  and  promote  the  growth  of  world-class  quality  and  best-value 
in  the  DOD  contractor  base. 


Once  operating  at  the  local  level,  business  area  teams  may  look  outside  their 
organization  to  form  wider  alliances.  The  real  benefits  of  this  program  are  only  realized 
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if  business  area  teams  are  formed  to  coordinate  their  business  processes  across  the  DOD 
components  and  present  “one  face”  to  their  industry  segment. 

Business  Area  Strategy 

Each  business  area  should  develop  a  Business  Area  Strategy.  This  includes  integrating 
the  business  area  in  a  coherent  strategy,  developing  goals  for  the  business  area,  and 
determining  uses  for  past  performance  information. 


Integrating  the  Business  Area 
Developing  Business  Area  Goals 
Determining  Uses  for  Past  Peiformance 
Information 


Integrating  the  Business  Area:  The  business  area  strategy  is  the  product  of  a  cross¬ 
cutting,  horizontal  integration  perspective.  It  starts  at  the  local  level  but  as  business  area 
alliances  are  formed,  it  becomes  more  robust  and  richer  and  exerts  a  DOD-wide 
influence.  At  the  peak  of  horizontal  integration,  it  becomes  the  backbone  for  a  “one- 
face”  to  industry  for  a  DOD-wide  business  are,  e.g.,  fights  aircraft,  engines,  a 
commodity  group,  medical  services. 


Through  its  unique  horizontal  integration  perspective,  the  business  area  strategy  wiU 
help  DOD  organizations  determine  how  to  meet  the  challenges  of  the  changing 
acquisition  environment  today  and  in  the  future.  Developing  a  business  area  strategy  is 
a  means  of  making  the  fundamental  departure  from  the  narrow  procurement  perspective 
of  the  use  of  past  performance  information  for  source  selection  decisions  to  a  broader 
business-like  viewpoint  expressed  below: 
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Developing  Business  Area  Goals:  From  our  analysis,  we  identified  goals  that  an 
effective  strategy  might  include: 

•  Develop  a  world-class  orientation 

•  Maintain  total  quality  with  focus  on  continuous  improvements 

•  Increase  the  number  of  high-quality  suppliers 

•  Improve  contributions  to  coiporate  profitability/operations 

•  Implement  a  team  approach  internally  and  extemally-new  suppliers  as  an 
integral  part  of  the  team 

•  Accredit  key,  critical  suppliers 

•  Develop,  coordinate,  commumcate  and  integrate  pricing  strategies  in  all  critical 
commodities 

•  Recognize  highly  reliable  sources  of  supply 

The  strategy  should  address  the  evaluation  of  contractor  performance  in  the  context  of  a 
total  program  tailored  to  the  particular  business  area.  Implementation  may  be  directed 
into  one  or  all  of  the  three  areas  of  the  Business  Area  Evaluation  Process:  measurement, 
certification,  and  improvement  The  results  of  the  strategy  process  may  be  that  only 
measurement  is  appropriate  for  some  products/services  whereas  more  aggressive 
certification  or  improvement  approaches  are  required  for  other  products/services. 

Determining  Uses  for  Past  Performance  Information:  The  strategy  elements  that 
relate  to  the  use  of  contractor  past  performance  information  in  contractor  selection 
decisions  introduce  the  need  for  a  tailored  approach.  Uses  for  contractor  past 
performance  information  may  include  any  or  all  of  the  following: 

[Tailor  solicitation/award  approach  to  selecting  contractors 

Make  secondary  decisions  once  long-term  contract  relationships  are 
established  (option  exercised  and  IDIQ  decisions) _ 

Manage  key,  critical,  strategic  suppliers  and  track  their  impact  on 
organization’s  performance  goals 


Recognize  superior  performance _ 

Build  long-term  relationship/partnerships 
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Achieve  specific  perfomriance  improvements  objectives 
‘  Avoid  incoming  quality  inspection 

-  Improvement  in  on-time  deliveries 

-  Enhanced  logistics  support _ 


Allocate  government  oversight  resources  commensurate  with  risk 


The  uses  of  past  performance  information  can  be  different  for  different  programs  and 
business  areas  and  should  be  addressed  in  the  strategy  at  the  business  area  level. 


Business  Area  Evaluation  Process 

Each  business  area  should  develop  an  evaluation  process  that  implements  the  Business 
Area  Plan  and  Business  Area  Strategy.  The  Business  Area  Evaluation  process 
establishes  the  methods,  steps,  and  procedures  for  collecting  and  evaluating  past 
performance  information  in  the  business  area.  The  business  area  evaluation  is  designed 
to  oisure  the  business  area  is  getting  die  informadon  to  select  world-class  suppliers  with 
a  best-value  outcome  with  the  goal  of  reducing  the  total  cost  of  doing  business. 


These  are  major  outcomes  of  a  business  area  evaluation  process: 


Measurement 


Certification 


^hngrovemen^ 


There  are  other  outcomes  that  may  be  more  appropriate  for  a  business  area.  These  are 
not  meant  to  be  mandated,  but  are  used  as  examples  generally  found  in  most  industry 
situations. 

The  business  area  evaluation  process  effort  is  a  challenge  to  adopt  the  elements  of  the 
measurem^t,  certification,  and  improvement  outcomes  that  are  appropriate  for  your 
business  area-Base/Camp/Post,  Central,  Systems,  or  Science  and  Technology. 
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Measurement:  Ongoing  performance  measurement  is  a  central  aspect  of  the  business 
area  evaluation  process.  The  benefits  of  performance  measurement  include! 

•  Systematic  collection  of  accurate,  relevant  data  for  contractor  selections 

•  Consistent  approach  to  measurement  across  rn^or  business  areas 

•  Consistent  feedback  to  contractors 

•  Focus  for  supplier  improvement 

•  A  tool  for  item  level  certification 

•  A  means  to  facilitate  benchmarking 

The  major  process  elements  for  developing  a  measurement  ^proach  are; 

_ Develop  Measurement  Criteria _ 

Develop  Approach  for 

_ Data  Validation  by  Contractors _ 

Develop  Performance  Feedback  Process 


Develop  Measurement  Criteria:  Each  business  area’s  business  area  evaluation 
process  defines  the  system  that  best  meets  its  business  re(]uirements,  while  incorporating 
common  criteria  for  measuring  supplier  performance.  Common  requirements  for  each 
business  unit  include: 

•  Development  of  a  procedure  detailing  (1)  rating  frequency;  and  (2)  rating 
communication  and  use 

•  Supplier  Performance  Report  with  ratings 

•  Minimum  reporting  frequency  of  quarterly 
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•  Distribution  policy  for  rating  results 

Incorporating  common  criteria  for  measuring  supplier  performance,  suppliers  are  rated 
on: 

•  Quality  of  the  products  and  services  they  provide 

•  Delivery  performance 

•  Ability  to  provide  service,  including  pertinent  information 

Each  business  area  must  define  what  constitutes  product  quality,  delivery  performance, 
and  service  requirements.  Measures  appropriate  to  die  business  area  then  need  to  be 
defined  for  the  three  rating  areas.  Quality,  delivery,  and  service  are  not  equally 
weighted  in  every  situation,  thus  there  is  a  need  for  each  business  area  to  devise  the 
appropriate  weights.  Each  business  area  may  weigh  the  three  categories  as  it  desires. 

Performance  measurement  is  used  to  select  contractors  to  do  business  with  and  to 
allocate  increased  or  decreased  business  to  a  contractor  based  on  performance  during  the 
current  contract  relationship.  This  could  occur  through  the  exercise  of  contract  options 
or  the  placement  of  contract  orders. 

Develop  Approach  for  Data  Validation  by  Contractors:  The  objective  of  this 
process  is  to  ensure  that  contractors  are  given  the  opportunity  to  review,  comment  on, 
and,  if  rqipropriate,  rebut  information  that  bears  on  their  performance  in  the  execution  of 
existing  contracts.  This  process  has  the  potential  to  be  used  in  awarding  future 
contracts. 

The  validation  process  will  typically  provide  contractors  with  access  to  information  that 
pertains  to  their  performance.  In  the  past,  this  has  been  accomplished  by  mailing  the 
performance  information  to  the  respective  contractors;  however,  the  performance 
tracking  systems  are  now  using  direct  electronic  access  for  this  purpose. 

As  information  moves  through  the  validation  process,  provisions  must  be  made  for 
distinguishing  between  validated  and  non-validated  data.  In  all  cases,  provisions  need 
to  be  established  for  retaining  any  comments  or  rebuttal  information  from  contractors 
relating  to  their  performance. 

Develop  Performance  Feedback  Process:  Feedback  is  provided  to  suppliers  on 
their  performance  results  to  improve  future  performance.  The  feedback  may  occur  in 
any  number  of  forms,  including: 
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•  In-person  meetings  with  suppliers  at  least  annually  to  inform  them  of  their 
evaluation  results,  identify  areas  of  improvement  and  develop  an  action  plan  for 
improvement 

•  More  frequent  notification  by  telephone  or  letter 

•  Notification  to  suppliers  of  their  performance,  and  their  performance  relative  to 
other  suppliers  for  their  product  or  service,  and  for  the  business  area  in  general 

Feedback  gives  both  parties  the  opportunity  to  improve  the  product,  reduce  costs,  and 
improve  service.  Maintaining  open  communication  helps  keep  contractors  informed  of 
their  performance  relative  to  all  contractors  within  their  commodity  groups.  Specific 
information  pertaining  to  a  single,  identified  supplier  is  never  shared  with  other 
suppliers. 


Certification:  Certification  to  the  item/part  or  family  of  parts  level  is  a  key  feature  of  a 
contractor  evaluation  process.  Certified  items  are  purchased  items  that  will  not  routinely 
be  subjected  to  incoming  inspection.  The  supplier  is  lesponsible  for  complying  to  form, 
fit  and  function  criteria  previously  evaluated  at  incoming  inspection.  Certification  is 
performed  on  an  item-by-item  basis.  When  certifying  a  conqionent,  the  specific  supplier 
manufacturing  location  that  is  producing  the  item  will  be  the  only  site  approved  to 
provide  the  certified  product 


Develop  Certification  Process  Procedures 
Identify  Key  Strategic 
Critical  Parts,  Materials,  Assemblies 
Conduct  Quality  System  Assessment 
__Develop  Total  Cost  Assessment 


Develop  Certification  Process  Procedures'.  The  following  should  be  given  due 
consideration  in  the  development  of  procedures  for  a  certification  process: 

•  Item  quality  level 

•  Financial  requirements 

•  Risk  analysis  of  using  a  certified  product 

•  Supplier  qualify  systems 

•  Supplier  process  capability 

•  Item  stability 


Procedures  should  also  address:  the  sharing  of  information  with  all  pertinent  parties,  i.e., 
business  area  management,  other  user  locations;  a  recognition  process  for  suppliers  of 
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certified  items;  review  changes  to  certified  items  or  the  process  in  which  they  are 
manufactured  to  ensure  the  change(s)  will  not  invalidate  the  original  qualification  for 
certification;  and  periodic  audits  or  reviews  to  determine  continuing  certification,  de¬ 
certification,  or  re-certification. 


The  criteria  for  and  risks  of  certification  will  be  detomined  at  the  irrunediate  business 
area  level.  Each  business  area  will  determine  die  minimum  amount  of  time  and  number 
of  defect-firee  receipts  that  are  acceptable  before  an  item  is  eligible  for  certification. 
Each  business  area  should  determine  a  suitable  threshold  risk  level  on  an  item-by-item 
basis.  The  risk  factor  will  vary  depending  on  die  supplier  plant  and  item  being  certified. 


A  quality  history  must  be  established  for  the  supplier  facility  producing  the  item  being 
considered  for  certification.  Historical  compliance  data,  i.e.,  supplier  delivery  and 
incoming  quality  performance,  quality  history  for  the  same  or  similar  item  produced  for 
another  facility,  and  supplier’s  product  complaint  levels  for  other  similar  products,  will 
help  validate  supplier  performance. 


Cotification  of  items  involves  site  visits  and  the  evaluation  of  processes.  To  provide  a 
thorough  understanding  of  the  supplier’s  process,  an  on-site  assessment  prior  to  item 
certification  is  essential.  Any  issues  found  during  the  assessment  must  be  resolved  prior 
to  certification  of  the  item. 

Identify  Key  Strategic  Critical  Parts,  Materials,  Assemblies:  Start  with  a 
manageable  number  of  critical  parts  and  expand  the  program  to  include  all  of  the  critical 
items  as  well  as  those  that  have  the  potential  to  reduce  operating  cost.  Part-level 
certification  is  vital  for  base/camp  port  and  central  commodities  business  areas. 


Conduct  Quality  System  Assessment  The  foundation  for  a  contractor  evaluation 
process  is  a  quality  systems  assessment.  Assessment  of  a  supplier’s  quality  system  can 
be  viewed  as  a  6-step  process. 
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This  is  a  sample  “Supplier  Quality  Process  Evaluation  Report”  resulting  from  a  quality 
system  assessment 
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This  particular  example  includes  both  a  quantitative  rating  and  a  narrative  section  to 
record  identified  improvement  actions  and  other  related  remarks. 


Eligibility  for  classification  as  a  certified  item  should  also  include; 

•  Responsibility  for  quality  lies  solely  with  the  supplier  of  an  item 

•  Regulatoiy  risk  requirements  must  be  reviewed  to  understand  impact  on  the 
certification  process 

•  Financial  risk  consideration  must  be  given  to  balancing  the  potential  risks  of  not 
routinely  inspecting  items  against  total  system  cost 

Develop  Total  Cost  Assessment  Effective  total  cost  assessments,  based  on  Activity 
Based  Costing  principles,  are  part  of  a  contractor  evaluation  program. 


Total  cost  encompasses  the  “all  in”  cost  of  doing  business  with  a  supplier.  Examples 
are:  acquisition  cost,  the  cost  of  supplier’s  activities  to  process  a  customer  order  and 
supplier’s  materials  plus  profit,  and  consumption  cost,  the  cost  of  a  customer’s 
activities— labor  and  overhead— to  process  a  supplier’s  shipment  through  the  customer’s 
system. 


The  objective  of  a  total  cost  assessment,  using  Activity  Based  Costing  principles,  is  to 
award  contracts  to  the  true  lowest  cost  bidder.  These  are  the  issues  to  be  addressed: 


•  the  historical  nonproductive  costs  resulting  from  supplier 
noncompliance 

•  the  estimate  of  the  true  cost  or  value  of  bids 

•  the  Suppliers’  nonproductive  events  that  are  “charged” 


Costs  of  Communications  I 

•  Source  rejection 

•  Inspection  resubmittal 

•  Return  to  supplier 

•  Material  review 

•  Shop  floor  rejection 
ilatentdefecij 

•  Corrective  action  request  letter 

•  Supplier  stop  notice 

•  CB;A  action 

•  Early 

•  Oversh^ment 

•  Late  receipt 

•  Paper  based  or  EDI 

Cost  Containment 

•  IfTventory  carrying 

•  Stability  of  pricing 
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Improvement:  The  improvement  aspects  of  a  business  area  evaluation  process  are 
aimed  at  evaluating  contractor’s  progress  in  achieving  the  highest  levels  of  performance. 
There  are  four  m^or  elements  in  the  improvement  process. 


Conduct  Process  Approvals 


Formulate  Pro  jects  to  Reduce  Cost/Improve  Quality 
Develop  On-Going  Cycle  of  Continuous 
_ Improvement _ 


_Dwdog^ImgrovementRocognitiOTftogam^ 


Conduct  Process  Approvels:  The  scope  of  processes  that  are  involved  in 
improvement-related  activities  is  more  extensive  than  the  certification  process  considers. 
Here  the  focus  can  extend  to  most  of  the  following: 

•  Quality/Service  History 

•  Cost  Management 

•  Environmental  Initiatives 

•  Quality  Systems 

•  Risk  Management 

Additionally,  management  and  technology  factors  are  considered: 

•  Management  commitment  of  the  business  area 

•  Industry  position 

•  Technology  position 

•  Resource  commitments  to  continuous  improvement 


The  Contractor  Processes  approval  is  contingent  upon  the  success  of  ongoing 
measurement  and  certification  efforts.  Those  processes  that  meet  minimum 
requirements  for  approval  will  be  approved. 


Three  possible  scenarios  for  approving  processes  are;  rely  on  eitiiCT  third-party 
certifications,  commercial  certifications,  or  both  (e.g.,  ISO  9000,  Malcolm  Baldridge 
National  Quality  Award  criteria,  or  other  commercial  certifications);  second,  grant  DOD 
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certification;  or  third,  either  include  third  party,  commercial,  DOD,  or  all  of  these  at  the 
time  of  each  acquisition— not  in  advance  as  implied  above.  This  would  potentially 
include  ISO  9000,  Baldridge,  or  other  commercial  certifications  in  addition  to  DOD. 


For  items  used  in  private  industry  for  which  DOD  has  a  need,  third  party  or  commercial 
certifications  may  be  appropriate.  The  administrative  burden  of  certification  would  be 
minimal  in  this  scenario.  In  other  situations,  where  the  item  is  unique  to  DOD,  DOD 
certification  may  be  appropriate.  This  scenario  may  requite  a  greater  burden-and  thus 
cost— to  administer. 

Formulate  Projects  to  Reduce  Cost/Improve  Quality:  Notwithstanding  previous 
^provals  for  processes,  projects  to  nuse  the  contractor’s  level  of  performance  can  be 
mutually  and/or  singly  identified.  In  this  role,  DOD  is  working  with  its  contractors  to  aid 
in  their  efforts  to  achieve  world-class  performance. 


Worldng  with  strategic,  oitical,  and  oth^  key  contractors  on  projects  to  upgrade  their 
performance  is  a  follow-on  activity  to  initial  certification.  The  goal  is  for  contractors  to 
achieve  Government  approval  of  processes  in  addition  to  any  parts  certification 
previously  achieved. 

Develop  an  On-going  Cycie  of  Continuous  Evaluation  and  Improvement  Self- 
assessments  and  performance  measurement  form  the  basis  for  the  continuous 
improvement  Process  evaluation  is  the  focus  of  continuous  improvement.  The 
progress  contractors  make  in  exceeding  their  initial  process  approval  levels  is  the  focus 
of  the  on-going  cycle  of  continuous  improvement  As  higher  performance  levels  are 
achieved,  new  targets  are  formed  and  progress  tracked  and  evaluated. 


The  best  cost  reduction  and  quality  improvement  results  will  be  obtained  from  steady, 
focused  continuous  improvement. 
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-  Cycle  of  continuous  evaluation  and  improvement 


Managing  value  into  both  parties’ 
operafions  on  an  ongoing  oasis. 

A  succe^l  program  incorporates 
the  supplier's: 

-  Management  commhment 

-  Quality/Service  history 

-  Cost  management 

-  Geography 

-  Environmental  Initiatives 

-  Quali^  systems 

-  Industiy  position 

-  Technology  position 

-  Risk  management  plans 

-  Resource  commitments 


► 


Eliminate  Saap 
Optimize  Packaging 
Increase  Delivefy 
Frequency 

Paperwofk^Adim,  Cost 
Reduction 
Inventoiy  Turns  & 
Investinent 

Product  Design  Review 
Process  Reviews 
Supplier  Development 
Deeper  into  the  Supply 
Cham 


The  respective  DOD  business  areas  should  be  proactive  in  all  aspects  of  flie  business 
area  evaluation  process,  but  particulaily  when  undertaking  continuous  improvetnent 
objectives.  The  “partnering”  model  is  the  posture  that  should  be  used  to  guide 
contractor  int^aces  in  the  improvement  environment 


Develop  Improvement  Recognition  Programs:  A  program  should  be  developed  to 
provide  feedback  and  to  recognize  accomplishments.  Documentation  describing  the 
business  area’s  evaluation  process  and  objectives  should  be  available  for  all  interested 
contractors.  Periodic  meetings  and  reports  should  be  a  part  of  the  program. 

Cost  The  cost  aspects  of  the  business  area  evaluation  process  track  the  level  of 
contractor  evaluation  the  business  area  has  adopted. 

•  Measurement— Cost  Assessment 

•  The  minimum  needed  to  support  best  value 

•  Certification-Total  Cost  Assessment 

•  Encompasses  the  “all  in”  cost  of  doing  business  with  a  supplier 

•  Improvement-Cost  Reduction 

•  Focuses  on  specific  target  opportunities  to  reduce  cost  using  activity 
based  costing 
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Activity  Based  Costing  Process  Fiow 

Procaduw  or  Tools  Utad 

Discussion 

Known  or  suspected  high-cost  and/or  non-value-added  activities 

Interviews,  observations,  personal  knowledge. 

Flowcharts 


Data  Coliection  Sheets 


Cost  Model  Worksheet 


Cost  Worksheet  Summary 


Procedure^policy  changes 


Comparison  of  Contractor  Evaluation  Program  and  IndusUy  and 
Government  programs:  The  Contractor  Evaluation  Program  model  is  used  in  this 
diagram  to  array  the  type  of  program  found  in  industry  and  DOD. 


Ford  -  Xerox  -  Baxter  Health  Care  -  British  Rail  -  Allen-Bradley  -Texas  i— 

Instruments  -  Boeing  -  Rockwell  -  McDonnell  -  Motorola  -  Allied  Signal  iinprovemeni 


National  Semiconductor  -  Loaer  -  Rsher  Scientific 


Certification 


Grainger  Distributors  -  Black&Decker 
Mobil  Oil  -  RJ.  Reynolds 


Measurement 


Mature  industry  programs  to  a  great  extent. 
Other  industry  and  government  programs  to 
a  far  lesser  degree  or  not  at  all. 


Business  Area  Plan 
and 

Business  Area  Evaluation  Strategy 
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3.  Functional  Requirements  for  the  Contractor  Evaluation  Program 

A  functional  requirements  document  for  the  Contractor  Evaluation  Program  was 
prepared  during  the  course  of  this  study.  Documentation  of  functional  requirements  for 
the  Contractor  Evaluation  Program  was  conducted  in  two  parts: 


Perform  a  functional  requirements  analysis.  We  analyzed  the  Contractor  Evaluation 
Program  model  to  identify  specific  functional  requirements  that  must  be  satisfied  by  a 
potential  information  system,  e.g..  Standard  Procurement  Systena,  Central  Contractor 
Registry.  We  then  identified  the  data  and  information  requirements  of  the  Contractor 
Evaluation  Program  model  that  must  be  satisfied  by  the  information  system. 


Develop  the  functional  requirements  document  This  is  a  statement  of  the 
functional  requirements  for  information  system  support  of  the  Contractor  Evaluation 
Program  model.  We  identified: 

•  Current  system  capabilities  that  need  to  be  retained 

•  Deficiracies  and  limitations  in  the  current  system  capabilities 

•  Additional  functional  and  performance  capabilities  that  will  be  required  to 
satisfy  new  or  changed  past  performance  requiremrats  from  die  Contractor 
Evaluation  Program  model 

•  Functional  and  performance  capabilities  that  provide  opportunities  for  increased 
economy  and  efficiency,  fiom  the  Contractor  Evaluation  Program  Model. 


We  implemented  a  Use  Case  Approach  to  analyze  and  document  functional 
requirements  for  the  Contractor  Evaluation  Program  model.  Through  Use  Case 
Analysis,  we  divided  the  Contractor  Evaluation  Program  model  into  a  collection  of  use 
cases.  Next,  textual  descriptions  of  each  use  case  were  developed  to  describe  the 
graphical  information  presented  in  each  cases. 


Once  the  functional  requirements  were  documented,  we  analyzed  the  Standard 
Procurement  System  functional  requirements  related  to  collection  of  contractor  past 
performance  information.  We  then  compared  them  to  the  functional  requirements  for 
the  Standard  Procurement  System  to  the  Contractor  Evaluation  Program.  The 
requirements  were  foimd  primarily  in  two  areas:  Administer  Contract  and  Procurement 
Planning. 
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Under  Administer  Contract,  it  indicated  that  the  system  shall: 

•  Notify  the  user  when  previously-identified  criteria  for  contractor  performance 
have  been  breached 

•  Process  material  review  board  actions  and  corrective  action  requests/  notices/ 
plans 

•  Track  contract  performance  reports 

•  Notify  the  user  when  performance  parameters  do  not  meet  user-defined  criteria 

•  Process  shipment  and  performance  data  against  the  MBLSTRIP  requisition 
number  and  contract  schedule. 


Under  the  Procurement  Plaiming  functional  requirements,  it  indicated  that  the  system 
shall  perform  a  Contractor  Assessment.  In  doing  so,  the  system  shall  automatically 

•  Aggregate  contract  performance  information  into  contractor  summary 
performance  reports 

•  Use  these  summary  reports  along  with  oth^  contractor  information  to  create 
vendor  rating  summary  reports. 

We  also  analyzed  the  Standard  Procuremrat  System  functional  requirements  related  to 
use  of  contractor  past  performance  information.  They  were  found  primarily  in  the 
Solicit  Offers  and  Award  Contracts  area. 


The  Solicit  Offers  and  Award  Contracts  section  it  indicated  that  to  evaluate  offers,  the 

system  shall  provide  die  capabili^  to: 

•  Evaluate  offers  based  on  the  offer  data  and  previously-defined  criteria 

•  Integrate  offer  data  and  previously-established  evaluation  criteria  to  perform 
evaluation 

•  Integrate  an  offeror’s  past  performance  information  into  the  evaluation  process, 
and  recommend  a  determination  of  responsibility  based  on  user-defined  criteria 
and  algorithms  applied  to  previously  entered  data 

•  Be  able  to  create,  request,  receive,  and  dispose  of  pre-award  survey  requests. 


Our  analysis  showed  that  the  Contractor  Evaluation  Program  model  functional 
requirements  are  consistent  with  apparent  Standard  Procurement  System  contractor  past 
performance  functional  requirements  from  the  standpoint  that  use  and  collection  criteria 
are  user-defined  to  the  Standard  Procurement  System, 
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4.  System/  Process  Issues  for  the  Contractor  Evaluation  Program 


The  Contractor  Evaluation  Program  implementation  must  consider  several 
system/pnocess  issues.  The  general  criteria  for  their  application  are  defined  by  the  three 
sections  of  die  Contractor  Evaluation  Program  and  the  relationship  between  the  specific 
system/process  i.e.  the  Business  Area  Plan,  Business  Area  Strategy,  and  Business  Area 
Evaluation  ftocess.  fii  two  cases-Faimess  and  Due  Process-tiie  issues  appeared  to  be 
related  more  to  the  Government’s  conduct  of  the  contractor  selection  process  and  how 
the  information  would  be  used  than  to  the  attributes  of  the  Contractor  Evaluation 
Program. 
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General  Criteru  for  Application 


Issues 

Business 
Area  Plan 

Business 

Area 

Strategy 

Business 

Area 

Evaluation 

Process 

Centralized  /  Decentralized 

X 

The  degree  and  level  of  centralization.  Is  data 
aggregated  to  the  product  center,  major 
command  or  HQ  level? 

Automated  /  Manual 

X 

Is  the  ^stem  or  process  automated,  semi- 
automated  or  manual  in  the  manner  in  which 
past  performance  information  is  collected, 
maintained  and  disseminated? 

Confidentialitv 

X 

System’s  capability  to  protect,  limit,  and 
otherwise  effectively  control  against 
unauthorized  access  to  contractor  past 
performance  data. 

Data  Availabilitv 

X 

System's  capability  to  r^idly  disseminate  the 
requested  standard  and  tailored  information  on 
real-time  or  time  delay  basis. 

Currencv  /  Inteeritv  /  Accuracv  /  Validity 

X 

AithirD  Little  so 


Issues 


GEf^RAL  CRITERIA  FOR  APPLICATION 

Business  Business  Business 

Area  Plan  Area  Area 

Strategy  Evaluation 
Process 


of  Data 


Cat^ories  of  data  included  in  the  system  -- 
government,  DOD  only  /  commercial. 
Quantitative,  qualitative  data.  Data  trends. 
Comparison  data. 


MergCTS  and  Acoiiisirinns 


System  capability  to  report  on  company's  past 
perforaiance  that  occurred  prior  to  a  merger  or 
acquisition. 


Subcontractor  Involvement 


The  system's  capability  to  discern  between  the 
prime  contractor  and  its  subcontractor's  past 
poformance  on  prior  contracts 


Fairness 

The  system's  capability  to  treat  all  offeror's 
equally  and  ^ure  past  performance  data  is 
evaluated  with  the  same  impartiality  as  other 
evaluation  data 


Due  Process 

Opportunity  for  contractor  to  respond  to 
weaknesses  or  deficiencies  documented  in  the 
government's  evaluation  process 


Lack  of  Past  Performance  Histo 


The  system's  capability  to  overcome  /  handle  the 
lack  of  past  performance  data  for  a  particular 
contractor. 


Threshold  of  ADolicabilit 


The  system's  capability  to  apply  data  according 
to  cost,  time,  or  oth^  (dollar)  thresholds. 


Cap^ilitv  of  Attribution 


The  system's  capability  to  shield  data  sources 
firom  unwarranted  disclosure. 


Contractor 

Selection 

Issue 


Contractor 

Selection 

Issue 
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GENERAL  CRITERIA  FOR  APPLICATION 


i 

Business 
Area  Plan 

Business 

Area 

Strategy 

Business 

Area 

Evaluation 

Process 

Consequences  to  the  Contractor 

X 

The  system's  intended  /  untended  penalties  / 
rewards  for  poor  /  superior  past  performance 
data 

Feedback 

X 

Recognition 

X 

The  system/process  issues  to  be  considered  under  the  Business  Area  Strategy  aspects  of 
the  Contractor  Evaluation  Program  are: 

•  Centralized/Decentralized 

•  Automated/Manual 

•  Data  Availability 

•  Source  and  Type  of  Data 

•  Lack  of  Past  Performance  History 

•  Consequences  to  the  Contractor 

•  Recognition 


The  system/process  issues  to  be  considered  in  the  Business  Area  Evaluation  Process 
aspects  of  the  Contractor  Evaluation  Program  are: 

•  Confidentiality 

•  Currency/Integrity/AccuracyA^alidity 

•  Merges  and  Acquisitions 

•  Subcontractor  Involvement 

•  Threshold  of  Applicability 

•  Capability  of  Attrition 

•  Feedback 

Specific  criteria  for  application  of  these  issues  should  be  developed  as  part  of  the 
respective  Business  Area  Strategy  or  Business  Area  Evaluation  Process  considerations 
to  which  they  relate. 
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SECTION  II.  D.  Business  Case  Analysis 


This  study  is  designed  to  assist  past  performance  policy  implementation,  not  systems 
development,  the  business  case  analysis  is  therefore  an  analysis  of  three  alternative 
policy  implementation  approaches  to  past  performance  policy  implementation.  The 
three  alt^^ves  that  will  be  used  for  comparison  purposes  in  the  business  case  analysis 
are  tte  “As-is”  model,  which  is  structured  from  the  information  in  Section  11.  A.  on 
existing  Government  past  performance  information  systems;  the  DFARS  model, 
developed  from  information  in  the  FAR  and  the  proposed  changes  to  the  DFARS-  and 
the  Contractor  Evaluation  Program  (To-be  model),  developed  from  information  in 
Section  IL  C  In  each  model  a  distinction  is  made  between  the  "Collection  of  Past 
Performance  Information  for  Future  Use",  and  the  "CoUection  and  Use  of  Past 
Performance  Information  during  Contractor  Selection".  The  analysis  of  and  comparison 
between  models  is  frxim  these  two  perspectives. 


1.  “As-ls”  Model 


The  “As-is”  model  for  existing  government  past  performance  information  systems,  as 
depicted  below,  reflects  the  situation  prior  to  recent  changes  to  the  FAR  and  the  related 
DFARS  case.  The  left  side  of  the  diagram  covers  the  collection  of  contractor 
performance  information  for  future  use.  The  right-hand  side  depicts  the  principal 

activities  performed  for  the  collection  and  use  of  past  performance  information  during 
contractor  selection. 
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Figure  IlD.l:  “As-is”  Model 

The  two  approaches  to  collecting  past  performance  information  for  future  use  that  are 
highlighted  in  the  center  of  the  diagram  are  ad  hoc  and  systematic.  The  ad  hoc 
collection  of  past  performance  information  for  future  use  has  been,  and  continues  to  be 
a  routine  practice  by  some  procurement  authorities.  The  primary  purpose  has  been  to 
support  local  contractor  selection  decisions.  In  most  cases  the  performance 
evaluations  are  not  provided  to  contractors  for  review  and  possible  rebuttal.  The 
evaluations  are  not  identified  as  "source  selection  information"  or  filed  for  possible  use 
in  the  future.  Information  from  these  activities  is  used  in  contractor  selection  decisions 
together  with  other  past  performance  information  that  may  be  gathered  at  the  time  of  a 
contractor  selection  decision. 

The  systematic  collection  of  past  perfcnmance  information  for  future  use  is  used  in  the 
systems  that  are  identified  and  described  in  Section  IL  A.  of  this  report.  These  existing 
systems  operate  in  essentially  two  different  ways  -  performance  tracking  and 
performance  appraisal.  Under  performance  tracking,  the  system  relies  on  the 
existence  of  performance  tracking  data  at  the  contract  line  item  level.  Under  this 
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system,  these  d3.ta  are  used  to  calculate  performance  ratings  based  on  previously 
established  decision  rules.  Typically  these  systems  address  attributes  of  supplies  and 
equipment  that  are  discernible  and  detected  at  the  time  of,  or  subsequent  to,  delivery 
by  the  contractor.  The  principal  focus  of  these  systems  is  on  the  quality  of  the  supplies 
and  equipment  and  the  timeliness  of  the  deliveries  by  the  contractor.  The  data  upon 
which  these  systems  rely  is  essentially  quantitative  and  objective  -  for  example, 
number  of  reported  defects  and  number  of  days  late  in  delivery.  The 
Red/Yellow/Gieen  and  the  Automated  Best  Value  hdodel  were  the  existing  systems 
that  used  this  approach.  The  process  analyses  and  the  automated  data  information 
system  analyses  for  these  two  systems  are  included  in  Appendix  B. 


Tht  performance  appraisal  systems  rely  on  the  appraisal  of  the  entire  contract. 
Appraisals  cover  the  work  performed  on  the  total  contract  or  contract  order.  Typically 
these  systems  address  not  only  the  quality  and  timeliness  of  products  delivered  by  a 
contractor,  but  also  additional  factors  dealing  with  the  performance  of  work  in-process 
and  overall  technical,  cost  and  schedule  performance  of  the  contractor.  These  factors 
might  include  any  one  or  all  of  the  following,  deprading  on  the  circumstances  of  the 
acquisition  and  the  nature  of  the  product  or  service  that  is  being  acquired; 

•  compliance  with  contract  requirements 

•  ovOTuns  experienced  on  reimbursable  contracts 

•  effectiveness  in  managing  the  provisions  of  the  contract  effectiveness  in  executing 
the  program  provisions  in  the  contract  (e.g.;  systems,  engineering  management, 
design  engineering,  manufacturing,  test  and  evaluation,  logistics,  subcontract 
management,  quality  assurance,  continuous  process  improvement,  etc.) 

•  the  quality  and  thoroughness  of  research  conducted  under  the  contract 

Our  analysis  identified  CPARS,  ACASS  and  CCASS  as  the  existing  systems  that  use 
this  approach.  The  processes  analyses  and  the  automated  data  information  systems 
analyses  for  these  systems  are  included  in  Appendix  C. 

The  right  side  of  the  diagram  for  the  “As-is”  Model,  depicts  the  activities  related  to 
the  collection  and  use  of  past  performance  information  at  the  time  of  contractor 
selection  and  contracting  decisions.  Market  research  has  been,  and  continues  to  be 
used  to  investigate  commercial  products  and  the  use  of  commercial  distribution 
systems.  In  addition,  market  analysis  is  also  undertaken  as  a  part  of  non-development 
item  initiatives.  There  is  no  specific  requirement  to  inquire  into  the  past  performance 
of  potential  sources  as  part  of  this  analysis.  Evaluation  of  an  offeror’s  past 
performance  is  encouraged  to  be  an  important  element  of  every  evaluation  and  contract 
award  for  commercial  items.  Likewise,  acquisition  strategies  and  plans  may  be 
formulated  in  anticipation  of  a  solicitation  for  certain  products  or  services.  There  is  no 
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stated  requirement  to  address  the  approach  for  dealing  with  contractor  past 
performance  as  a  part  of  these  strategies  and  plans. 


The  activities  at  the  bottom  right  of  the  As-Is  Model  diagram  are  performed  in 
connection  with  gathering,  validating  and  using  past  performance  information  at  the 
time  of  a  contractor  selection  decision.  These  activities  are: 

•  verify  past  performance  information  furnished  by  contractors  in  response  to  the 
requirements  in  the  solicitation; 

•  access  validated  past  performance  information  for  use  in  source  selection,  to  the 
extent  that  it  may  exist;  and 

•  gather  and  assess  past  performance  information  from  ott^r  sources,  such  as 
information  available  from  risk  assessments,  process  reviews,  government 
maintained  databases,  performance  award  listings,  and  commercial  survey  services. 

•  utilization  of  past  performance  information  in  contractor  selection  decisions  based  on 
the  ground  rules  established  for  the  acquisition  and  consistent  with  die  evaluation 
criteria  and  odier  information  provided  to  the  offerors. 


2.  DFARS  Model 


The  DFARS  model  is  depicted  in  the  diagram  on  the  next  page.  It  is  based  on  our 
interpretation  of  die  DFARS  Case  at  this  time.  With  respect  to  the  collection  of  past 
performance  information  on  active  contracts  for  future  use  poformance  evaluations 
will  essentially  be  required  on  all  contracts  above  $100,000  with  few  exceptions  and 
results  of  the  evaluation  will  be  provided  to  the  contractors  for  their  review,  comment 
and  possible  rebuttal.  Methods  for  handling  the  review  process  and  resolving  any 
differences  between  contractor  and  government  officials  are  also  covered  in  the  model. 


With  respect  to  the  collection  and  use  of  past  performance  information  at  the  time  of 
contractor  selection,  the  DFARS  includes  the  requirement  to  use  contractor  past 
performance  information,  except  in  those  cases  where  its  use  is  not  found  to  be 
practical  or  useful.  In  those  cases,  the  contracting  officer  must  document  in  the 
contract  file  the  reasons  why  past  performance  was  not  used. 


ArthirD  Little 


66 


Collection  of  Past 
Performance 
Information  for 
Future  Use 


Evaluate  and  record 
performance  of 
oontractors  on 
active  contracts 


t 

Provide  contractors  «rfth 
access  to  perfonnance 
Information  for  review 
and  rebuttal 

Resolve  Issues 
raised  by  contractors 
and  file  formation 
for  future  use 


Performance  evaluation  and  validation 
Gfthis  information  is  required  on  an 
active  contracts  in  accordance  with 
standard  criteria  and  rfota  elements 
•  The  FAR  states  that  the 
requirement  should  be  talored  to 
the  size,  conterd  and  oomplmity 
of  the  contractual  raquirarnent. 
•fiut.  standard  evaluation  criteria 
and  common  data  aiemertis  are 
prescrfbed  for  use  by  the  draft 
DFARS  In  evaluating  an  corfoacts. 
•CQnsequtntty.fte  amount  of 
"taiioring*  is  ganemtiy  nmitad  to  an 
entry  of  ^applicable* 


^  unvaiktelBd  informatton  ■ 

- — - -  vandstsd  Informatfon 


^  vafidfllsdlnforinetton 


Collection  and  Use  of  PastPerfonnan!^ 
Information  During  Contractor  Sel^ion 


Conduct  market 
research  and  analysis 
on  procurement 
reiyilrements 


Develop  acquisition 
strategies  and  plans 
forpnxurement 
regukements 


Prepare  contractor  past 
performance  pfovi^s 
for  inclusions  in 
solicttations 


Verify  past 
performance 
^formation  furnished 
by  contractors 


Access  validated 
past  performance 
information  for  use 
in  source  selection 


No  specific 
requiremerttto 
address  contractor 
past  performance 
issues  ki  market 
research  and 
acquisition  planning 
SAME  AS  THE 
•AS-IS“MODa 


Specific  lequirements 
^  have  been  astabkshed 
for  using  contractor  past 
partormance  hformation 
In  ths  source  selection 
process.  If  not  used, 
the  contracting  officer 
must  document  reasons 
why  in  the  contract  file. 

Utilize  past 
performance 
kformation  In  source 
selection  decteions  • 


Gather  and  assess 
past  performance 
Wormatioo  from 
other  sources 


Figure  nD.2:  DFARS  Model 


A  significant  point  concerning  the  DFARS  model  is  that  although  the  FAR  states  that 
the  requirement  for  the  evaluations  should  be  tailored  to  the  size,  content  and 
complexity  of  the  contractual  requirement,  the  DFARS  (as  of  the  current  draft) 
establishes  standard  evaluation  criteria  and  common  data  elements  to  be  used  in  all 
cases.  Consequently,  the  amount  of  "tailoring"  could  be  constrained  by  the  use  of  the 
standard  criteria.  The  following  table  provides  a  summary  of  these  provisions  of  the 
FAR  that  indicate  that  the  collection  and  use  of  past  performance  information  can  be 
tailored  to  the  particular  circumstances  of  the  procurement. 
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Collection  of  past  pefformance 
Infonnation  for  ftiture  use 
(ML  FAR,  SUBPART  42.15) 

Collection  and  use  of  past  performance 
information  at  the  time  of  source  selection 
(Ref.  FAR,  PARTIS) 

The  content  and  fomiat  of 
performance  evaluations  shall  be 
established  in  accordance  with 
agency  procedures  and  should  be 
tailored  to  the  size,  content  and 
complexity  of  the  contractual 
requirements. 

[Ref.  42.1502  (a)] 

The  cognizant  technical  officiai  Is  respon^ble  for  the  technical  and 
past  performance  requirements  related  to  the  source  selection 
process.  [Ref.  15.604(b)J 

Past  performance  shall  be  evaluated . . .  unless  the  contracting  officer 
documents  in  the  contract  file  the  reasons  why  past  performance 
should  not  be  evaluated.  [Ref.  15.605  (b)  (1)  (ii)] 

The  source  and  type  of  p£^  performance  information  to  be  included  in 
the  evaluation  is  within  the  broad  discretion  of  agency  acquisition 
officials  and  should  be  tailored  to  the  circumstances  of  each 
acquisition.  [Ref.  15.608  (a)  (2)  (ii)] 

The  FAR  provisions  provide  government  officials  the  latitude  to  develop  and  adopt  a 
tailored  approach  that  fits  the  specific  circumstances  of  each  acquisition.  On  the 
other  hand,  the  proposed  DEARS  provides  criteria  and  a  rating  scheme  that  shall  be 
used  in  the  evaluation  of  contractor  performance.  These  prescriptions  could  be 
constraed  to  limit  the  extent  of  tailoring  that  may  be  attempted. 


The  evaluation  criteria  and  rating  scheme  in  the  DEARS  is  summarized  on  the  next 
page. 


AithirD  Little 


68 


EVALUATION  AREAS  AND  FACTORS 

RATINGS 

Quality  of  Product  or  Service  (a  required  element).  This  includes  thi 
following  aspects  of  performance: 

1.  Compliance  with  contract  requirements; 

2.  Accuracy  of  reports; 

3.  Appropriateness  of  contractor  personnel  assigned  to  the  contract 

a  Unsatisfactory:  Nonconformances  compromise 
(or  are  compromising)  the  achievement  of  contract 
requirements,  despite  the  use  of  Agency  resources 
Marginal :  Nonconformances  require  major 

Agency  resources  to  ensure  achievement  of 
contract  requirements. 

Satisfactory :  Nonconformances  do  not  impact 
achievement  of  contract  requirements. 

Excellent:  There  are  no  quality  problems. 

cost  Control  (not  required  for  firm-fixedi)rKe  and  firnvfb(ed>price  with 
economic  price  adjustment  contracts).  This  Includes  the  following  aspec 
of  performance; 

1.  Current  accurate,  and  complete  WHIngs; 

2,  The  relationship  of  negotiated  cost  to  cictuals; 

3,  Cost  containment  initiatives;  and 

4.  The  number  and  cause  of  change  orders  Issued. 

Unsatisfactory :  Cost  Issues  are  compromising 
s  perforrnance  of  contract  requirements. 

Marginal :  Cost  issues  required  (or  require) 

Agency  resources  to  ensure  achievement  of 
contract  requirements. 

Satisfactory:  issues  do  not  impact 

achievement  of  contract  requirements. 

Excellent:  There  are  no  cost  issues. 

Timeliness  of  Performance  (a  reqt^red  element).  This  Includes  the 
following  aspects  of  performance: 

1 .  Whether  the  contractor  met  Interim  milestones; 

2.  Contractor's  responsiveness  to  technical  direction; 

3.  Contractor's  responsiveness  to  contract  change  orders  and 
administrative  requirements; 

4.  Whether  the  contract  was  completed  on  time,  including  contract  clos 
out  and  reporting  responsibilities  and  contract  administration;  and 

5.  Whether  liquidated  damages  were  assessed. 

Unsatisfactory:  Delays  are  compromising  the 
achievement  of  contract  requirements,  d^ite  the 
use  of  Agency  resources. 

Marginal:  Delays  require  Agency  resources  to 
ensure  achievement  of  contract  r^uirements. 
Satisfactory:  Delays  do  not  Impact  achievement 
j  of  contract  r^uirements. 

Excellent:  There  are  no  delays. 

Contracting /Business Relations  (a discrotionary elemenQ.  This 
Includes  the  following  aspects  of  performance; 

1.  Whether  the  contractor  effectively  rnanaged  the  contTa:t  effort; 

2.  How  responslvo  the  contractor  was  to  contract  re<^rements; 

3.  How  proaipdy  the  contractor  notified  the  Govemrnento^ 

4.  Whether  the  contractor  was  reasonable  and  cooperative; 

5.  How  flexible  the  contractor  was; 

.6.  Was  the  contractor  proactive; 

7.  How  effective  were  contractor  recommended  solutions;  and 

8.  Did  the  contractor  effectively  Implement  socioeconomic  programs. 
Including  compliance  with  requirements  of  the  clause  of  FAR  52.218 
8,  Utilization  of  Small  Business  Concerns  and  Small  Disadvantaged 
Business  Concerns,  and  52J219-9,  Small  Business  and  Small 
Disadvantaged  Business  Subcontracting  Plan. 

Unsatisfactory:  Response  to  Inquiries.  lechnIcaJ 
service,  and  administrative  issues  is  not  effective 
and  responsive. 

Marginal:  Response  to  Inquiries,  technlcaf 
service,  and  administrative  issues  is  marginally 
effective  and  responsive. 

Satisfactory;  Response  to  Inquiries,  technical 
service,  and  administrative  issu^  Is  usually 
effective  and  responsive. 

Excellent:  Response  to  Inquires,  technical 
service,  and  administrative  Issues  is  effective  and 
responsive. 

Figure  IID  J:  DFARS  Past  Performance  Evaluation  Criteria 


There  may  appear  to  be  some  advantages  in  a  single  set  of  evaluation  criteria  for  all 
contracts,  just  as  there  are  apparent  disadvantages  that  bring  into  question  a  “one  size 
fits  all”  approach  to  evaluating  contractor  past  performance.  These  questions  are 
addressed  later  in  this  section. 


3.  Contractor  Evaluation  Program  Model  CTo-be”  Model) 


The  diagram  below  incorporates  the  Contractor  Evaluation  Program  described  in 
Section  11.  C.  The  principal  difference  between  the  Contractor  Evaluation  Program  and 
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the  “As-Is”  and  DFARS  models  is  the  focus  on  business  areas ,  shown  in  the  shaded 
area  in  the  center  of  the  diagram.  This  emphasis  on  business  areas  includes:  a  business 
area  plan,  a  business  area  strategy  to  guide  the  collection  and  use  of  past  performance 
information;  and  a  business  area  evaluation  process,  tailored  to  the  specific  requirements 
of  the  business  area. 


Figure  IID.4:  Contractor  Evaluation  Prr^ram  Model 

The  principles  that  were  used  in  designing  the  model  were  derived  from  an  analysis 
of  our  previous  research  in  this  area  and  from  the  government  and  industry 
approaches  to  contractor  past  performance  and  supplier  evaluation  that  we  reviewed 
in  this  study.  They  included  the  following: 

•  A  cost-effective  approach  to  the  collection  and  use  of  contractor  past  performance 
information  depends  on,  and  is  sensitive  to  factors  related  to  the  business  areas  in 
which  products  and  services  are  procured  and  used  (as  opposed  to  a  universal 
approach  that  can  be  applied  to  the  full  range  of  products  and  services  procured  by 
DOD  in  all  sectors  of  the  industry). 

•  A  business  area  consists  of  a  homogeneous  group  of  products  or  services  which 
share  similar  characteristics  and  for  which  a  forward-looking  plan  and  a  coherent 
and  congruous  strategy  and  evaluation  process  can  be  developed. 
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•  Business  areas  can  be  local  or  extended  in  application.  In  their  most  robust  form, 
they  constitute  the  horizontal  integration  of  products  and  services  across 
organizational  lines. 

•  The  process  for  implementing  contractor  past  performance  issues  in  a  particular 
business  area  is  developed  from  business  area  plans  and  strategy  for  the  specific 
business  area  and  typically  involves  a  cross-functional  team  effort. 

•  The  initial  and  vital  step  in  developing  plans  and  strategy  for  a  business  area  is  an 
analysis  that  covers  the  requirements  for  the  product  or  service,  past  and 
projected;  the  industry  composition  and  basis  of  competition;  and  the  market 
trends  and  specific  performance  of  leading  companies  in  the  industry. 

•  The  business  area  plan  and  strategy  will  provide  the  basis  for  developing  a 
tailored  approach  to  the  collection  and  use  of  contractor  past  performance 
information  in  the  particular  business  area  as  well  as  the  foundation  for  a  total 
program  designed  to  incorporate  best  value  practices  into  the  procurement 
process  and  to  attract  contractors  and  suppliers  committed  to  high  levels  of 
performance. 

•  Information  technology  will  be  utilized  to  facilitate  communication  between 
Government  managers  in  separate  mganizations  with  a  need  to  share  information 
about  business  area  strategies  and  plans  as  well  as  the  past  performance  of 
individual  contractors  in  those  business  areas. 

The  key  elements  of  the  Contractor  Evaluation  Program  as  shown  on  the  top  level  of 

the  accompanying  diagram  are: 

•  A  Business  Area  Plan; 

•  A  Business  Area  Strategy  in  the  context  of  an  overall  acquisition  strategy  for  tiie 
business  area 

•  A  Business  Area  Evaluation  Process  that  can  be  used  to  tailor  the  contractor  past 
performance  provisions  of  the  FAR  and  the  DFARS  to  the  particular  business 
area. 


Each  of  these  elements  is  interrelated  and  are  shown  at  three  levels  of  detail  in  the 
diagram  below. 
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The  Business  Area  Plan  at  the  second  level  of  the  diagram:  defines  the  business  area 
to  be  investigated;  includes  a  research  input  using  internal  and  indusby  data;  and 
presents  an  analysis  of  all  the  internal  and  external  factors  related  to  the  business 
area. 

The  Business  Area  Strategy  at  the  second  level  of  the  diagram  includes : 

•  Integration  of  contractor  past  performance  issues  with  the  broader  issues  related 
to  the  overall  acquisition  program  for  the  business  area,  and  with  the  cross¬ 
functional  considerations  that  may  be  involved  (e.g.,  engineering,  test  and 
evaluation,  production,  logistics,  risk  management,  and  quality) 

•  Goals  for  the  contractor  past  performance  program  in  the  business  area,  including 
the  desired  level  of  performance  sought  from  contractors  with  whom  contracting 
relationships  exist  or  are  anticipated 

•  Approach  for  the  use  of  past  performance  information  designed  to  achieve  the 
best  value  and  world-class  suppliers  goals  and  objectives  established  for  the 
business  area,  and  to  provide  guidelines  for  developing  a  cost  effective  plan 
tailored  to  the  particular  business  area. 

The  Business  Area  Evaluation  Process  is  designed  to  lay  out  a  plan  for  executing  the 
Business  Area  Plan  and  Strategy.  Three  principal  areas  are  identified  for  possible 
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coverage  in  the  process,  these  may  be  changed  or  others  may  be  added  when 
warranted  in  a  specific  business  area. 


The  three  principal  areas  are: 

•  A  process  to  measure  die  performance  of  contractors  based  on  criteria  tailored  to 
the  business  area  and  consistent  with  the  strategy  for  the  business  area^ 

•  A  process  to  certify  the  performance  of  contractors  consistent  with  the  strategy 
for  the  business  area  and  that  considers  the  certification  of  processes,  products 
and  services  (e.g.,  "Blue  Ribbon"  Programs);  and 

•  A  process  to  improve  the  performance  of  contractors  that  constitute  the  supplier 
base,  in  a  manner  consistent  with  the  strategy  established  for  the  business  / 
program  area  (e.g.,  process  improvement  initiatives  and  recognition  programs). 

Whereas  all  processes  should  address  the  performance  measurement  aspects  of  the 
program,  the  other  two  areas  will  be  coveted  to  the  extent  that  the  agreed  strategic 
approach  provides  guidance  and  direction  in  these  areas. 

The  third  level  of  the  diagram  addresses  tiie  m^faods  that  will  be  used  to  measure 
past  performance  information  on  contractors  that  participate  in  the  business  area. 

The  following  methods  are  candidates  for  use; 

A  continuous  measurement  program  tiiat  Q^ically  may  include  provisions  for  review 
of  the  information  by  the  contractor;  a  process  for  dealing  with  the  resolution  of 
contractor  rebuttals;  and  the  maintenance  of  the  information  for  future  use  when 
needed  for  source  selection  purposes,  or  some  other  purpose  consistent  with  the 
strategic  plan.  This  approach  is  implemented  in  either  of  the  two  following  ways: 

•  Periodic  performance  appraisals  at  the  contract  or  contract  order  level  based  on  an 
assessment  by  the  responsible  government  official(s)  for  contract  technical  and 
management  oversight;  and 

•  Continuous  performance  tracking  at  the  contract  line  item  level  based  on  quality  and 
delivery  data  collected  as  a  part  of  established  contract  management  and  oversight 
processes. 


An  ad  hoc  measurement  program  that  is  designed  to  provide  past  performance 
information  when  needed  to  support  contractor  selection  decisions,  or  some  other 
purpose  established  in  the  strategic  plan.  This  type  of  program  may  include  information 
from  sources  such  as: 

•  Surveys  of  prior  customers-e.g.,  reference  checks; 

•  Requests  for  past  performance  information  fi’om  contractors  (e.g.,  in  response  to 
solicitations); 
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•  On-site  assessments  of  contractor  operations  to  include  their  technical  and 
"management  processes;  and 

•  Inform^on  gathered  from  other  available  source  (e.g.,  product  performance  and 
reliability  data,  C/SCS  data,  certifications  and  awaixls,  etc.). 


The  ^ntr^tor  Evduation  Program  is  a  comprehensive  approach  for  collecting  and 
providing  informatron  on  the  past  performance  of  contractors  for  contractor  selection 
purposes.  It  is  also  an  orderly  approach  to  tailoring  policy  and  requirements  to 
specific  business  areas,  and  to  achieving  and  sustaining  inqrrovements  in  the  overall 
level  of  performance  exhrbited  by  contractors  in  the  business  area. 


4.  Difference  Between  Contractor  Evaluation  Program  and  DEARS  Models 

The  following  summarizes  the  principal  differences  between  the  Contractor  Evaluation 
Program  and  the  DEARS  model  in  dealing  with  contractor  past  performance  issues, 
poUcies  and  requirements.  Each  of  the  criteria  indicated  in  the  chart  is  discussed  in  the 
following  paragraphs. 
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DIFFERENCES 


CRITERIA 

{T - - - - - - - - - - 

DFARS 

Contractor 

Evaluation 

Program 

Flexibility 

Organizations  are  provided  latitude  and  empow  - 
ered  to  tailor  requirements  and  guidelines  to  fit 
particular  characteristics  of  their  business  ara^ 

Limited 

Substantial 

Scope 

Consideration  of  past  performance  information 
extends  beyond  its  use  in  soutice  selection 
decisions  to  include  improvement  initiatives 

None 

Significant 

Business  areas  Focus 

An  analysis  of  business  areas  is  recognized  as  a 
key  factor  in  developing  effective  strategies  and 
plans  for  dealing  with  past  performance  fosues 

No 

Yes 

Process  integration  &  Teamwork 

Past  performance  strategies  and  plarrs  are 
developed  consistent  with  overall  acquisition 
strategies  and  utilizing  cross-functional  teamwork 

Limited 

Yes 

VALUE  TO  THE  USER 

Information  on  the  past  perfonnance  of  contractors 
provides  a  valuable  Input  to  sources  selection 
dedsions  and  is  shared  with  other  omanizations 

Umited 

Substantial 

SHARING  Information 

Provisions  are  made  for  sharing  contractor  past 
performance  Information  among  IDOD 
organizations  in  a  cost  effective  manner 

Not  addressed 

Yes 

Figure  IID^:  Comparison  of  DFARS  and  Contractor  Evaluation  Program  Models 
Flexibility 

The  Flexibility  criterion  addresses  the  capability  to  deal  with  and  adapt  to  the  particular 
circumstances  of  an  acquisition  program.  It  is  especially  important  in  the 
implementation  of  past  performance  policy  and  requirements  within  the  DOD  because 
of  the  wide  range  of  products  and  services  that  are  acquired  and  the  wide  range  of 
circumstances  that  may  affect  the  acquisition  process  leading  up  to  the  selection  of 
contractors  and  to  the  award  of  contracts.  In  addition,  the  post-award  activities  and  the 
contract  management  approach  used  by  DOD  components  is  also  subject  to  considerable 
variability  depending  on  factors  such  as  the  size,  scope,  complexity,  and  nature  of  the 
contracted  work. 

Both  models  are  designed  to  address  the  flexibility  criterion.  The  extent  of  the 
flexibility  in  the  Contractor  Evaluation  Program  is  considerably  greater  than  the  FAR  / 
DFARS  model,  as  summarized  in  the  following  table. 
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CONTRACTOR  EVALUATION _ 

Organizations  with  contracting  authority  and 
technical  oversight  responsibilities  shall  establish 
the  content  and  format  of  performance  evaluations 
based  on  analysis  of  their  business  areas  and 
consistent  with  strategies  and  processes 
established  by  these  organizations. 

Contractor  evaluations  will  be  tailored  to  the 
business  areas  in  which  the  contracts  are  issued 
and  to  the  requirements  established  in  the 
ccmtracts,  This  tailoring  may  extend  to  the 
thresholds  used  and  to  the  provisions  for  review 
and  rebuttal  by  the  contractor. 

Principal  users  of  contractor  past  performance 
infcrmation  include  government  officials  involved  in 
contracting  decisions.  Users  of 
the  information  shall  have  a  major  role  in 
determining  the  scope  and  content  of  contractor 
evaluations  in  specific  business  areas. 

Same  as  the  FAR  /  DFARS  model,  except  that 
cognizant  technical  officials  will  also  ensure  that 
their  responsbilities  are  discharged  in  a  manner 
consistent  writh  the  strategy  and  the  plan 

established  for  the  business  area. _ 

Same  as  the  FAR  /  DFARS  model,  except  that  the 
contractor  past  performance  strategy  and 
implementing  process  shall  cover  guidelines  and 
pro>^  decteion  rules  for  determining  the 
inclusions  of  past  performance  factors  in  source 
selection  and  contracting  decisions. 

Same  as  the  FAR  /  OFARS  model,  except 
the  source  and  type  of  past 
performance  information  is  first  tailored  to  each 
business  area  and  the  approach  is  described  in  the 
strategy  for  the  business  area. 


_ FAR /DFARS  MODEL _ 

Content  and  format  of  performance  evaluations 
shall  be  established  in  accordance  with  agency 
procedures  and  should  be  tailored  to  the  size, 
content  and  complexity  of  the  contractual 
requirements.  (Ref.  FAR  42.15) 

Contractor  evaluations  will  be  prepared  on  all 
active  contracts  above  the  $100,000  threshold, 
reviewed  by  contractors  subsequent  to  the 
evaluation,  and  filed  and  protected  as  ‘source 
selection  information*  after  resolution  of  any 
rebuttal  by  the  contractor.  (Ref.  FAR  42.1 5) 

The  evaluation  of  contractor  performance  shall 
include  specific  data  elements  and  evaluation 
areas,  factors  and  ratings  (as  delineated  in  the 
proposed  DFARS  42.15) 


The  cognizant  technical  official  is  responsible  for 
the  technical  and  past  perfonnance  requirements 
related  to  the  source  selection  process.  [Ref.  FAR 
15.604(b)] 

Past  performance  shall  be  evaluated  (in  contract 
award  decisions) . .  .unless  the  contracting  officer 
documents  in  the  contract  file  the  reasons  why  past 
performance  should  not  be  evaluated.  [Ref.  FAR 
15.605  (b)] 

The  source  and  type  of  past  performance 
information  to  be  included  in  the  evaluation  is 
within  the  broad  discretion  of  agency  acquisition 
officials  and  should  be  tailored  to  the  circumstances 
of  each  acquisition.  [Ref.  FAR  15.608  (a)] 


Figure  nD.6:  Contractor  Evaluation  Program  and  FARS/DFARS  Flexibility 

Differences 

A  more  rigid  structure  for  the  collection  and  use  of  contractor  past  perfonnance 
information  would  very  likely  simplify  the  information  processing  functions  and  the 
automated  systems  that  may  be  used  to  support  these  functions.  Our  analysis  indicated 
that  the  value  of  past  performance  information  to  the  user for  contractor  selection 
purposes  diminishes  as  the  degree  of  standardization  is  increased  in  the  evaluation 
process  and  in  the  collection  of  information.  The  views  of  the  government  officials  who 
were  interviewed  during  the  course  of  the  project  and  who  partidpated  in  the  workshops 
tended  to  support  this  view. 


Scope 

The  Scope  criterion  addresses  the  coverage  each  model  provides  for  all  types  and 
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sources  of  contractor  past  performance  information  and  for  all  potential  uses  of  this 
information.  Provisions  in  Subpart  15.8  and  in  Subpart  42.15  of  the  FAR,  supplemented 
by  the  provisions  in  the  proposed  DFARS,  address  the  types  and  sources  of  past 
perfomance  information  and  one  purpose  served  by  this  information  (i.e.,  for  source 
selection  purposes).  The  statement  in  the  FAR  Subpart  15.8  states: 

“. . .  the  solicitation  shall  afford  offerors  the  opportunity  to  identify . . .  contracts 
performed  by  the  offerors  that  were  similar  in  nature  to  the  contract  being  evaluated,  so 
that  the  Government  may  verify  the  offerors’  past  performance  on  these  contracts. 

Past  performance  information  may  also  be  obtained  from  other  sources  known  to  the 
Government,  "nie  source  and  type  of  past  performance  information  to  be  included  in  the 
evaluation  is  within  the  broad  discretion  of  agency  acquisition  officials  and  should  be 
tafiored  to  the  circumstan^  of  each  acquisition.  Evaluations  of  contractor  performance 
prepared  in  accordance  with  Subpart  42.15  are  one  source  of  performance  information 
which  may  be  used.” 


Whereas  supplemental  FARS/DFARS  guidance  could  be  provided  to  address  other 
types  and  sources  of  past  performance  information  and  the  potential  use  of  this 
information  for  other  purposes,  none  is  currently  available.  The  OFPP  guide  on  best 
practices  for  past  performance,  published  in  May  1995,  is  recognized  to  be  an  interim 
measure.  Some  of  the  guidance  provided  in  this  document  does  not  appear  to  have 
relevance  to  DOD  acquisition. 


The  Contractor  Evaluation  Program  is  intended  to  provide  a  broader  perspective  to 
contractor  past  performance  and  is  designed  to  address  a  total  systems  approach  to  the 
collection  and  use  of  past  performance  information,  to  include: 

•  The  type  of  analyses  required  to  develop  a  tailored  approach  by  business  area 

•  A  definitive  strategy  for  dealing  witii  the  entire  issue  of  contractor  past  performance 
in  each  business  area  in  a  maimer  consistent  with  the  overall  acquisition  strategy  and 
procurement  planning  for  the  business  or  program  area;  and 

•  A  process  focused  on  the  actions  necessary  to  execute  the  contractor  past 
performance  strategy  in  areas  such  as  performance  measurement,  product  and 
process  certification,  and  performance  improvement  initiatives. 

Coverage  in  the  Contractor  Evaluation  Program  is  provided  for  the  performance 
appraisal  information  that  is  addressed  in  Subpart  42. 15  of  the  FAR,  and  performance 
tracking  systems,  such  as  the  Navy’s  RedA^ellow/Green  system  and  DLA's  Automated 
Best  Value  Model.  The  proposed  model  also  covers  past  performance  information 
related  to  the  certification  of  contractors  for  products  and  services  as  well  as  the 
processes  employed  by  the  contractors. 
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Business  Area  Focus 

This  aiteiion  addresses  the  c^ability  to  effectively  deal  with  the  size,  scope  and 
diversity  of  the  DOD  acquisition  program. 


The  proposed  Contractor  Evaluation  Program  recognizes  that  the  products  and  services 
acquired  by  DOD  span  from  sophisticated,  multi-million  dollar  we^>on  systems  to 
relatively  simple,  inexpensive  commodities.  Using  FY  ‘94  data,  the  following  table 
illustrates  the  size  of  the  DOD  procurement  program  as  well  as  die  range  of  products 
and  slices  diat  are  acquired.  Also  shown  is  the  breakdown  of  the  total  dollars  into  the 
various  cat^ories. 
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Research  &  Development 


Communitv  Service 


Defense  Systems 


Defense  -  Other 


Environmental  Protection 


General  Science  &  T< 


Medical 


Natural  Resources 


21.824 


isas 

JlSE 

UtiH  _ 


Mechanical  Power  Transmission 


Bearinas 


Woodworkinq  Machinery  and 


Metalwofldnq  Machine 


Service  and  Trade  E 


Machine 


IKDl 


IBacwi 

ESE 


iiEiBgg 


Furnace  /  Steam  Eauiir  &  Nudear  Reedors 


Other  Services  &  Construction 


iili 


Auto.  Data  Processinq  &  Telecom.  Senrices 


Purchase  of  Structures  and  Fadlrties 


Natural  Resources  Management 


43.946 


Ibi 

_ lEi 

mirn 


Values 


Mamtenance  and  R 


Measurinq  Tools 


Hardware  and  Abrasives 


Quality  Control.  Tesdna  and 


Maintenance.  R 


Modification  of  E 


Technical  Representation  Services 


Eit3aa!r.!Ai'H.n,wm 

EE 


iTrnrTi' 


and  Veneer 


Materials 


. . . 

ME  m 
laissii-iMussi 
ME 

_ 

Biiz: _ 

ME 

ME 

ME _ 

FMI.’.HffilrliUlf, 


EDEIjSjainEaEEniOErsS 


IKEll 

IKEll 

iKnl 

iKnl 


IKSlI 


mal 


Instruments  and 


Lease  or  Rental  of  Eouioment 


Lease  or  Rental  of  Fadlitias 


Construction  of  Structures  and  Facfllties 


Marnt.  Repair  or  Afteration  of  Real 


mpi 

|Rnfc?hiM;<T 

IE 


Chemicals  arxj  Chemical  Products 


Trarninq  Aids  and  Devices 


Food  Preparation  and 


Office  Equip.,  Text  Process.  /  Visible  Reryntte 


Supplies  and  Equipment 


liElCSSSiSS 

IlD^ 
liBi 
lEI 

Ie 


62.342 


Nudear  Ordnance 


Fire  Control  Equipment 


Ammunition  and 


Guided  Missiles 


Aircratt  Launch.  Landi 


Space  Vehides 


,  Small  Craft.  Pontoons  /  Floatinq  Docks 


Ship  and  Marine  Eaui 


E“ - 

Ei 

EBSS 

EEffla  _ 

El 

*siUl3BBi5MPn'!!icS5?I?!MrffiBiiu« 

Ei 


|E 

Ie 

lEi 

Iei 
Jebi 


Books.  Maps,  and  Other  Pubiications 


Musical  Instruments 


Recreational  and  Athletic 


Cleaning  Equipment  and  S 


Brushes.  Paints.  Sealers  and  Adhesives 


CoTTtainers, 


Textiles,  Leather.  Furs.  Apparel,  Tents/ 


Vehicular  Equipment  Com 


Tires  and  Tubes 


~z:^P!!!gL 

mi - 

g^lKClB 

S^Vee 

EE 

lEltl 

mi 

likjE 


IICTi 


I  Kill  I 


Toiletries 


Aqncwtural  Supplies 


Live  Animals 


Subsistence 


Fuels.  Lubricants.  Oils,  and  Waxes 


Nonmetallic  Fabricated  Materials 


Nonmetailic  Crude  Materials 


Metal  Bars,  Sheets  and 


Ores.  Minerals  arxl  Their  Primary  Products 


Miscellaneous 


Figure  IID.7:  FY1994  Product/Service  Acquisitions  ($M) 


AithirDLHtle 


79 


Included  in  the  tabulation  shown  in  the  table  above  are  all  contract  actions  above 
$25,000.  The  total  of  these  actions  was  about  $1 18  billion  in  FY  ‘94,  which  was 
divided  mto  about  19%  for  R&D,  37% 

for  services  and  construction,  and  44%  for  supplies  and  equipment.  The  total  DOD 
procurement  program  for  FY  94  accounted  for  about  67%  of  all  federal  departments 
and  agencies. 


It  is  important  to  understand  the  size  and  scope  of  the  DOD  procurement  program  as 
well  as  the  number  of  organizations  that  have  procurement  authority  and  technical 
oversi^t  responsibilities  for  a  portion  of  the  total  program.  Some  of  the  principal 
organizations  in  the  major  DOD  components  are  listed  below.  In  addition,  contracts 
are  awarded  by  the  operational  organizations  in  each  service  including  bases,  posts  and 
camps. 


Amnm«n«,Mun«ons»Chem.Com.Headquarte(»U.S.MailneCo(p  OWahomaCKy  Air  UgMcs  Center 

AimamenlR&OCenter  MiUtaiy  Seem  Command  Ogden  Air LogMics Canter 

Chamicat  Biological  Defense  Com.  Offce,  Ctilef  Naval  Reseanch  Sactamento  Air  Logistica  Center 
Aviation  and  TiDop  Command  Strategic  Systems  Pfognam  Office  Sen  Anlorib  Air  togWict  Center 
Comnamications  A  Bect!oniesColTu^tevai  Air  Systems  Comi^  Warner  Robins  Air  Logistics  Center 

MMeCommand  Space  &  Naval  Warfare Sys.  Com.  Aerospace Quidanee and  Metrology 

Tank-Automotive  Command  Naval  Fadfltios  Engineering  Com.  Center.  OH 

Troop  Support  Command  Naval  Sea  Systems  Command  Space  and  Mssile  Systems  Center 

Ballistic  Research  Latoratory  Regional  Contracting  Ctr..  Wash.  Becbonic  Systems  Center 

Research  Laboratory  Regional  Contracting  Ctr..  Phfl.  Aaronautlcat  Systems  Canter  (ASC) 

Belvoir  Research  Dev.  A  Eng.  Ctr.  Regional  Cont  Ctr..  Long  Beach  Directorate  of  RAD,  ASC 
Defense  Supply  Servica  Wash.  ♦  Navy  Aviation  Supply  Office  Air  f=orce  Dev.  Test  Center,  Egfin 

MStary  Traffic  Management  Com.  «  Navy  Ship  Parts  Control  Center  Air  Education  A  Training  Com.,  TX 

Medical  ROAAL  Command  a  Mare  Island  Naval  Shipyard  Air  Mtttaiy  Command,  Scott  AFB 

Health  Services  Commands  a  Pearl  Hartxx  Naval  Shi^rd  Air  Combat  Command.  Langley,  VA 

Brooke  Medical  Center  a  Portsmouth  Naval  Ship^  10th  Air  Base  Wing,  USAF  Academy 

Central  Contracting  Office  *  Philadelphia  Naval  Sl^yaid  Air  Inteigence  Agency 

Eisanhovver  Medical  Center  a  Norfolk  Naval  Shipyard  Air  Force  Space  Com.,  Peterson  AFB 

Fltzsimorts  Medical  Center  Submarine  Support  Fadfity,  Groton  Air  FotcaReserve,  Robbins  AFB 
Beaumont  Medical  Center  a  Air  Warfare  Center.  Lakehurst  Space  and  Missile  System  Center,  LA 

Madigan  Medical  Center  Air  Warfare  Center,  Warminster  HQ  Space  Command,  Peterson  AFB 

Tripier  Medical  Center  a  Air  Warfare  Center,  Indianapolis  Human  Systems  Center,  Brooks  AFB 

Walter  Reed  Medical  Center  Air  Warfare  Center,  Pax.  River  Armstrong  Laboratory,  Brooks  AFB 

U.S.  Aimy  Depots  Air  Warfare  Center,  China  Lake  Phillips  Ljaboratory,  Kirtland  AFB 

Anniston  Depot  a  Air  Warfare  Center.  Ortando  Wright  Laboratory,  WP  AFB 

Corpus  Christi  Depot  a  Surface  Warfare  Center,  Crane  Frel^  Test  Center,  Edwards  AFB 

Letterkenney  Depot  Surface  Warfare  Ctr.,  Indian  Head  Rome  Air  Development  Center 

Red  River  Depot  Surface  Warfare  Ctr.,  Panama  City  Arnold  Eng.  Dev.  Ctr.,  Arriold  AFB 

Tobyhanna  Depot  Surface  Warfare  Center,  Dahlgren  Civfl  Engineering  Center,  Tyndall  AFB 

Tooele  Depot  Surface  Warfare  Center,  Carderock  Office,  Scientific  Resear^  Bolling  AFB 

Corps  of  Engineers  *  Surface  Warfare  Center,  Louisvfile 

Information  Systems  Command  ♦  Undersea  Warfare  Ctr,,  Keyport 
info.  Sys,  Selection  &  Acquisition  Undersea  Warfare  Ctr.,  Newport.  Rf 
Exchange  Senr.  Com.,  Va,  Beach 
Oceanographic  Office,  Stennls,  MS 

♦Navy  activities  that  use.  or  Command,  Control  A  Ocean  Surv.  Ctr. 
have  used  the  RYG  system  Headquarters,  Naval  District  Wash. 

«  Fleet  &  industrial  St^.  Ctis.  (9  sites) 

Construction  Organizatior^  (6  sites) 


Defense 
Contracting 
Mgmt  Coalman 


DLA 

(Contracting) 


-  General  Supply 
Center,  Richmond 

“Construction  Supply 
Center,  Columbus 

"  Electnonics  Supply 
Center,  Dayton 

^  Fuel  Supply  Center, 
Alexandria,  VA 

“  Industrial  Supply 
Center,  PhHa. 

-Personnel  Support 
Center,  Phlla. 

-Subsistence  Region 
Pacific.  Alameda 

■  ADP/T  Contractirrg 
Office,  Alexandria 

■  National  Stockpile 
Center,  Arl.  VA 


Figure  IID.8:  Principal  Organizations  That  Acquire  Products/Services 
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Under  review,  the  DOD  acquisition  program  dwarfs  anything  in  the  commercial  world. 
Even  the  largest  commercial  opmtions  are  relatively  small  by  comparison.  And  most 
of  the  major  firms  focus  their  business  in  a  relatively  few  areas  (e.g.,  automobiles, 
software,  aircraft,  etc.). 


The  tremendous  size,  scope  and  diversity  of  the  DOD  acquisition  program,  as 
indicated  in  the  preceding  discussion,  represented  a  significant  challenge  during  the 
course  of  conducting  the  study  and  examining  the  contractor  past  performance  issue. 

It  was  found  that  any  discussion  or  analysis  of  contractor  past  performance  required  a 
quahfying  statement  diat  established  the  particular  segment  of  the  total  program  being 
addressed.  For  example,  a  method  tfiat  was  reasonable  and  logical  for  one  segment  was 
found  to  be  irrelevant  or  even  counter  productive  in  other  areas.  These  observations 
provided  the  basis  for  one  of  the  design  characteristics  that  has  been  built  into  the 

Contractor  Evaluation  Program  - ,  the  capability  to  deal  with  the  inherent  differences  in 
the  various  segments  of  the  DOD  acquisition  program. 


The  business  area  focus  in  die  proposed  model  is  achieved  by  addressing  each  business 
area  as  a  separate  entity,  analymg  die  factors  relevant  to  the  business  area,  and  then 
devising  a  strategy  that  makes  soise  for  dealing  with  contractor  past  performance  at  die 
business  area  level.  This  information  is  then  used  to  develop  ground  rules  and  to  devise 
a  process,  for  considering  past  performance  in  contracting  decisions,  and  improving  the 
overall  performance  of  contractors  in  a  particular  business  area. 


The  FAR  /  DFARS  model  does  provide  for  tailoring  but  does  not  address  the  idea  of 
using  the  business  area  analysis  as  the  basis  for  devising  a  sensible,  cost  effective 
approach  to  contractor  past  performance.  The  FAR  gives  some  recognition  to  market 
research  in  Part  1 1  for  determining  the  availability  of  commercial  products  in  the 
marketplace  for  Government  use.  The  Non-Developmental  Item  (NDI)  handbook  does 
address  market  investigations  for  NDI  purposes.  Neither,  however,  enconqiasses 
contractor  past  performance  considerations,  nor  is  it  undertaken  on  a  continuing  basis. 


Process  Integration  &  Teamwork 

This  criterion  deals  with  provisions  for  handling  the  integration  of  past  performance 
considerations  with  other  factors  and  analyses  that  may  be  pertinent  to  a  particular 
business  area,  or  to  the  acquisition  program  or  programs  that  constitute  the  business 
area.  The  criterion  also,  encompasses  the  teamwork  and  coordination  needed  for 
dealing  with  the  cross-functional  interests  and  perspectives  that  may  be  a  factor  in  the 
larger  more  complex  business  areas. 
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The  process  integration  and  teamwork  provisions  in  the  Contractor  Evaluation  Program 
are  primarily  addressed  in  the  development  of  a  business  area  strategy.  This  activity 
also  contributes  to  the  process  by  which  the  vertical,  overall  acquisition  strategy  is 
developed  for  a  program  area. 


At  the  present  time  the  DEARS  model  does  not  spedfically  address  the  integration  nor 
die  teamwork  aspects  for  dealing  with  contractor  past  performance,  either  in  the 
collection  of  information  on  contractor  performance  or  in  the  use  of  this  information  for 
contractor  selection  purposes.  Guidance  in  this  area  could  be  developed  and  provided  in 
a  separate  document,  there  does  not  appear  to  be  recognition  in  the  DFARS  for  an 
integrated  and  cross-functional  teaming  approach  to  the  implementation  of  the 
contractor  past  performance  policy. 


Value  to  the  User 

Hiis  oiterion  deals  with  the  capability  to  focus  on  the  needs  of  the  ultimate  user  and  to 
provide  past  performance  inform^on  that  has  value  to  the  users  —  government  officials 
involved  in  die  acquisition  of  products  and  services. 


The  Contractor  Evaluation  Program  is  designed  to  ensure  that  it  will  provide  useful 
information  to  users  by  incorporating  the  following  features: 

•  Users  define  the  specific  evaluation  criteria  to  be  used.  This  feature  is  based  on 
one  of  the  underlying  principles  embedded  in  the  Contractor  Evaluation 
Program;  namely,  use  States  collection.  The  specific  approach  for  handling 
the  past  performance  contractors  is  developed  at  the  business  area  level.  Part  of 
this  process  provides  for  developing  specific  ciiteria  that  will  be  used  to  evaluate 
the  performance  of  contractors.  The  primaiy  usct  of  this  information  is  the  same 
organization  that  collects  the  information,  or  that  oversees  its  collection. 

•  Users  meuntain  local  files  on  contractor  past  performance.  When  past 
performance  information  is  gathered  on  a  continuing  basis  for  future  use  by  a 
particular  business  area,  it  will  typically  consist  of  either  performance  appraisal 
information  or  performance  tracking  information.  Performance  appraisals  are 
generated  locally  by  government  officials  with  contract  management  oversight 
responsibility.  And  tracking  information  is  typically  gathered  from  separate 
databases  that  cover  quality  and  delivery  performance.  In  either  case,  this 
information  is  available  across  the  business  area  and  continuously  updated  for 
future  use  in  the  selection  of  contractors  to  perform  similar  work. 
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By  specifying  top  and  second  level  criteria  the  DFARS  model  limits  the  extent  to  which 
the  needs  of  the  ultimate  users  of  the  information  are  consideied. 

Past  Performance  Information  Sharing 

This  criterion  deals  with  the  capability  to  share  past  performance  information  among 
government  organizations. 

h  the  Contractor  Evaluation  Program,  provisions  are  made  for  sharing  two  types  of 
information  within  the  business  areas  -  administrative  and  specific  past  petformance 
for  use  by  other  organizations  in  a  business  area..  This  information  could  be  tqipended 
to  a  Central  Contractor  R^istry  or  similar  centralized  system,  through  a  lead  site  within 
the  biKiness  area,  or  it  could  be  provided  separately.  The  following  is  a  brief 
description  of  each  type: 

•  The  administrative  information  includes  on-line  access  to  the  full  range  of  data 
from  the  Federal  Procurement  Data  System  (FPDS)  for  any  product  or  service 
code  of  interest.  These  data  include  the  identity  of  contractors  that  provide 
various  products  and  services  to  toe  government,  including  contract  numbers, 
types,  and  dollar  value.  Additional  information  would  also  include:  a  synopsis 
of  TOntract  work  statements;  an  indication  if  past  performance  information  was 
available  for  a  particular  contractor  at  a  certain  location;  the  availability  of 
planning  information  for  the  particular  business  area;  and  contact  points  for  toe 
purpose  of  obtaining  additioi^  information  and  coordinating  with  toe  other 
government  organizations. 


•  The  past  perfomumce  information  would  include  information  available  at  other 
business  area  sites,  based  on  criteria  used  by  the  business  area.  This  information 
would  be  accessible  by  direct  contact  with  the  other  organizations  by  whatever 
means  are  established  by  toe  organization  that  maintains  the  information  (e.g., 
telephone,  e-mail,  FAX,  and  database  access).  Provisions  would  ensure  that 
access  is  provided  only  to  authorized  users. 
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Section  III.  Conclusions 


niis  section  is  organized  into  three  parts.  The  first  is  designed  to  provide  our  responses 
to  three  fundamental  questions  DUSD  (AR)  needs  to  consider  to  implement  past 
performance  policy  implementation.  The  second  part  states  general  lessons-leamed  that 
we  believe  should  guide  past  performance  policy  implementadon.  The  third  presents 
additional  associated  conclusions. 


A.  Responses  to  DUSD  (AR)  Questions 


Question:  Should  DOD  collect  and  use  the  past  perfonnance  Information 
required  by  FAR  Part  42? 

Answer: 

Yes,  became: 

•  It  makes  good  bminess  sense  as  proven  by  overwhelming  indmtry  acceptance 

•  It  is  being  med  successfully  in  DOD  now,  although  on  a  very  limited  scale 

•  It  can  be  tailored  to  fit  specific  circumstances,  although  proposed  DFARS 
policy  has  the  effect  of  limbing  tailoring. 

Discussion: 

We  reviewed  government  and  industry  experience  to  answer  the  question  of  whether  to 
irr^lement  FAR  Part  42. 


There  is  a  very  limited  amount  of  Government  experience  available.  The  past 
performance  information  systems  in  use  in  DOD  acquisition  accounted  for  a  very  small 
percentage  of  DOD  actions  or  dollars.  They  were  not  a  valid  resource  for  answering 
this  question  for  aU  the  different  types  of  products/services.  However,  the  commodity- 
based  systems  we  observed  were  successfully  meeting  their  intended  purposes. 


There  is  extensive  industry  supplier  evaluation  program  experience  (the  industry  parallel 
to  our  definition  of  a  Government  past  performance  information  system),  that  supports 
the  implementation  of  FAR  Part  42  guidance  by  DOD.  However,  there  is  very  little 
industry  experience  with  services  and  major/small  system  acquisitions  using  supplier 
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evaluation  program  techmques.  There  is  also  an  extensive  amount  of  industry 
experience  on  ^proval  of  processes  in  addition  to  measuring  contract  performance. 


The  benefits  of  establishing  ongoing  performance  measurement,  certification,  and/or 
approval  programs  support  implementing  FAR  Part  42  requirements.  These  benefits 
include; 

•  Improved  service 

•  Decreased  costs  (transportation,  product/part,  transaction  (labor),  payment  terms, 
inventory,  operations) 

•  Increased  quality 

•  Improved  development  time  to  introduction  of  new  technologies 

•  Increased  customer  satisfaction  and  loyalty 

•  Higher  employee  morale 


The  process  essentially  leverages  the  information  that  would  otherwise  be  collected  for 
contractor  selections  by  using  it  for  performance  measurement/feedback,  certification, 
and/or  improvement  programs. 


Questions: 

(a)  What  information  shouid  be  coilected? 

(b)  What  type  of  approach  should  be  used? 

(c)  What  direction  and  guidance  shouid  be  provided? 

Answer  (a): 

The  information  that  is  collected  should  be  driven  by  its  planned  uses  and  tailored  to 
the  business  area  requirements.  The  business  area  planning  and  strategy  steps  will 
help  define  the  planned  uses  for  past  performance  information  and  top-level  criteria, 
e.g.,  Quality,  Cost,  Delivery,  and  Service.  The  next  level  of  definition  should  be 
accomplished  in  the  business  area  evaluation  process. 


Answer  (b): 

The  DOD  approach  should  follow  these  general  principles: 

Decentralized,  Focus  on  Business  Areas,  Total  Program  Context,  Horizontally 
Integrated,  User  and  Use-Driven,  Simple,  and  Share  InforrtwtiotL 
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Discussion: 

An  explanation  of  these  principles  follows: 

•  Decentralized— The  range  of  products  and  services,  and  variance  in  size,  scope, 
type,  and  complexity  of  contracts  makes  a  standard,  DOD-wide  system 
in^ractical.  Government  and  industry  expmence  support  a  decentralized 
approach  supported  by  general  guidelines,  decision  rules,  best  practices,  and 
information  technology  support 

•  Focus  on  Business  Areas-The  iiiq)lementation  of  past  performance  should 
focus  on  individual  business  areas  at  die  operating  level  which  encompass 
similar  products  or  services  from  which  a  coherrat  and  congruous  strategy  can 
be  developed  by  organizations  with  procurement  authority  and  technical 
responsibility. 

•  Total  Program  Context— Past  performance  needs  to  be  viewed  in  the  context  of 
a  total  program  that  goes  beyond  die  collection  and  use  of  past  performance 
information,  and  covers: 

•  Analysis  of  individual  business  areas  including  both  internal  and  external 
factors 

•  Development  of  a  sensible  strategy  for  contractor  past  performance  at  the 
business  area  level. 

•  Processes  designed  to  implement  the  strat^y  for  business  areas  in  which 
the  organization  is  active 

•  Horizontally  Int^ated— The  business  area  concept  starts  at  the  local  level, 
where  it  is  integrated  with  the  acquisition  strategy  and  procurement  plaiming  for 
the  business  areas.  As  business  area  alliances  are  formed,  they  exert  a  DOD- 
wide  horizontal  integration  effect  by  joining  similar  business  areas  across  the 
DOD  components.  The  implementing  direction  needs  to  emphasize  the  need  for 
integration  and  coordination. 

•  User-Driven-The  users  of  past  performance  information  need  to  have  the 
principal  role  in  defining  what  information  to  collect,  when  to  collect  it,  and  how 
to  make  it  available  for  their  use  in  selecting  contractors.  The  users  should 
include  the  technical,  management,  and  procurement  officials  who  are  involved 
in  and  responsible  for  making  contractor  selection  decisions. 

•  Simple— To  be  effective,  the  past  performance  approach  has  to  be  easy  to 
understand  and  explain,  without  bemg  simplistic,  or  it  runs  the  risk  of  being 
misunderstood,  ignored,  or  both. 
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•  Share  Information— Systems  and  processes  for  sharing  past  performance 
information  among  organizations  depend  on  all  of  the  above  and  should  be  dealt 
with  after  all  of  the  above  are  dealt  with. 


Answer  (c): 

DUSD(AR)  past  peifomtance  implementation  policy  and  guidance  should  consider 

the  following  specif  conclusions: 

•  Past  performance  policy  implementation  should  follow  the  tenets,  procedures, 
and  techniques  of  tiie  Contractor  Evaluation  Program  or  a  similar  program, 

•  The  past  peifomtance  information  collection  requirements  of  FAR  Part  42 
should  be  implemented  for  commodity  acquisitions,  except  for  commercial 
products, 

•  The  past  performance  implementation  requirements  of  FAR  Part  42  should  be 
tested  on  a  pilot/prototype  basis  for  the  acquisition  of  services. 

•  The  past  peifomtance  informative  requirements  of  FAR  Part  42  should  not  be 
required  for  major/smaU  systems.  A  pQot/prototype  system  should  be  tested  for 
mqjor/smaU  systems,  witii  emphasis  on  (he  evaluation  of  processes. 


Discussion: 

Based  on  the  industay  and  Government  information  available,  it  appears  reasonable  for 
DOD  to  implement  FAR  Part  42  collection  requirements  for  commodity-type 
acquisitions. 


There  is  very  little  data  for  service-type  acquisitions  other  than  A&E  and  construction. 
Although  the  Corps  of  Engineos  systems  appeared  to  be  successful  for  their  highly- 
tailored  application,  we  do  not  feel  comfortable  endorsing  that  specialized  experience 
for  all  the  different  services-types  procurements. 


For  major  systems,  CPARS  is  a  successful  system.  However,  it  was  not  clear  there  is  a 
benefit  to  collecting  vast  amounts  of  contract-related  past  performance  information  over 
a  six-  to  ten-year  timeftame  fw  a  systems  acquisition  when  it  may  not  be  used  for  a 
similar  contractor  selection  until  years  later.  The  benefits  may  be  more  forthcoming 
from  collecting  process  related  past  performance  information. 
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Question:  How  should  information  be  collected~Single  system,  Decentralized 
systems,  Ad  hoc  only? 

Answer: 

There  is  no  single  answer.  The  response  is  sub-divided  into  three  perspectives  -  single 
system,  decentralized  systems,  or  ad  hoc. 

Single  System: 

A  single  DOD-wide  past  performance  information  system  would  be  effective  only  as  a 
**red  and/or  blue  flag”  system  and  could  not  provide  the  detailed  analyses  to  support 
best-value,  world-class  contractor  selection. 

Discussion: 

The  weight  accorded  past  performance  information  in  contractor  selections  has 
considerably  increased.  Contractor  past  performance  evaluation  generally  has  been 
conducted  in  a  very  dicumscribed  manner,  coo^ared  to  what  is  envisioned  in  the  new 
policies: 

•  Past  performance  has  bera  a  minor  factor  in  source  selection 

•  Typically  valued  at  5-15% 

•  Generally  not  a  discriminator  in  the  selection  process. 

•  Past  performance  information  collection  generally  has  been  ad  hoc  and 
concurrent  with  the  source  selection  process 

•  Forms/Calls  to  program  managers  and  contracting  officers  are  made  on  current 
and  expired  contracts 

•  Contractor  submissions  are  in  response  to  RFPs,  RFQs 

•  Contract  management  performance  measurement  data  was  not  designed  to  be 
used  in  source  selection 


Because  of  the  weight  that  is  plaimed  to  be  given  past  performance  information,  the 
accuracy,  detail,  and  relevance  of  past  performance  information  must  be  the  highest 
level,  in  order  to  support  the  selection  of  world-class  suppliers  on  a  best  value  basis.  It 
is  very  doubtful  that  a  single  system  for  the  hundreds  of  thousands  of  transactions  that 
occur  annually  (and  possibly  millions  over  a  three -year  period  for  maintaining  the  data) 
could  provide  the  relevant  and  detailed  past  performance  information  that  is  required. 
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The  problem  of  accuracy  poses  another  considerable  issue.  With  few  exceptions,  the 
accuracy  of  quality  and  delivery  source  data  in  legacy  systems  is  inadequate  to  support 
widespread  use  of  past  performance  information  systems.  Previous  attempts  by  the 
Services  have  failed  because  of  the  lack  of  accurate  data  from  source  databases.  Data  of 
the  highest  reliability  is  needed  for  a  credible  past  performance  information  system. 

The  RedATellow/Gieen  system  has  been  successful  because  of  Ihe  reliability  of  the 
quality  performance  data  in  its  source  databases.  It  has  been  providing  contractor 
quality  performance  ratings  for  over  five  years.  However,  its  efforts  to  expand  to 
include  ratings  for  delivery  performance  were  slowed  by  the  lack  of  reliable  delivery 
data.  Die  introductiori  of  new  past  performance  systems  is  dependent  on  reliable  source 
databases,  without  which  new  past  performance  information  systems  caimot  be  used. 

We  see  the  process  as  one  in  which  the  data  to  be  collected  und^  FAR  Part  42,  should 
be  designed  to  produce  accurate,  relevant,  and  detailed  data  needed  for  the  new  past 
performance  information  systems. 


Low 


Data 

Quality 


y  High 

July  1 995  January  2000 


DecentraU'Zfid  System: 

Decentralized  systems  organized  on  a  business  area  basis  are  the  most  cost-effective 
approach  to  implementing  FAR  Part  42  collection  requirements. 
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The  reasons  that  led  us  to  develop  the  Contractor  Evaluation  Program  model  are  the 
same  that  support  this  conclusion.  A  decentralized  system  with  a  business  area  focus 
will  provide  the  relevant,  detailed  data  to  support  best-value,  world-class  contractor 
selection  decisions.  Over  time  accuracy  will  be  improved,  but  procedures  are  required 
to  insure  quality  data  is  collected  and  provided  to  selection  officials. 


Ad  hoc  Approach: 


This  approach  may  not  provide  the  systematic,  accurate,  relevant  data  that  is  needed 
to  support  past  peiformance  as  a  major  factor  in  contractor  selection. 


Systematic  collection  of  past  performance  data  is  essential  to  supporting  the  increased 
weight  being  accorded  past  performance  in  contractor  selection.  Due  to  the  distributed 
nature  of  the  data  that  is  collected  ad  hoc,  it  is  highly  unlikely  that  it  will  be  entirely 
relevant,  accurate,  or  detailed  enough  to  support  the  prominent  evaluation  weight  being 
accorded  to  past  performance  in  contractor  selections.  This  ad  hoc  approach  is 
characteristic  of  the  maimer  in  which  past  poformance  data  is  collected  today. 
Improvements  being  made  by  DOD  and  odier  Federal  agendes  in  collection  techniques 
are  a  step  in  the  right  direction.  However,  until  procedures  are  added  for  contractors  to 
validate  die  data,  it  is  difficult  to  imagine  the  ad  hoc  sqpproach  as  being  fully  crqiable  of 
supporting  weighty  past  performance-based  contractor  selection  decisions. 
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B.  Lessons  Learned 


The  following  lessons  learned  are  provided  for  DUSD  (AR)  consideration: 

•  Achieving  full  implementation  can  take  5  to  7  years.  Nonetheless,  the 
perspective  and  its  evolving  development  offers  vision,  a  road  map,  and 
confumation  of  direction  for  DOD. 


•  Lmking  past  performance  strategies  to  overall  acquisition  reform  strategy  and 
initiatives  is  critical 

•  Partnering  and  multi-functional  teamwork  at  all  levels  with  internal  customers 
and  suppliers  are  essential. 

•  All  stakeholders  must  be  identified  and  explicitly  considered  in  process 
improvements. 

•  Information  systems  and  accurate  data  are  critical  to  implementation. 

•  The  challenge  to  improve  the  supplier  base  is  difficult,  but  can  be  achieved  by 
working  with  siq>pliers  in  a  win-win  relationship. 

C.  Associated  Conclusions 


There  are  a  number  of  barriers  in  the  DOD  environment  to  widespread  adoption  of  past 
performance  as  a  major  selection  factor  in  a  best-value  context: 

•  Low  bidder  mindsetfculture 

•  Risk  avoidance  culture 
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•  Lack  of  experience  with  subjective  decisionmaking 

•  Need  to  educate  buyers  that  there  is  a  choice 

•  Time  to  validate  performance  information 

•  Weak/inaccurate  quality  and  delivery  data  processes 

•  Lack  of  tools  to  collect  accurate  data 

•  Impact  on  acquisition  streamlining  efforts  to  reduce  procurement  administrative 
lead  time  (PALT) 

•  Productivity  Impact 

•  Administrative  burden 


Innovative  change  management  training  programs  will  be  required  to  meet  past 
performance  policy  implementation  and  its  related  world-class  supplier  and  best-value 
objectives. 

•  Cultural  change  to  support  other-than-”low-cost”  mentally  is  slow  to  take  place 
without  new  learning,  team  enviromnrat,  management  commitment,  and  sound 
automation  systems. 

•  Industry  supplier  evaluation,  performance  measurement,  and  recognition  of 
successes  and  techniques  may  need  to  be  introduced. 

•  Government  initiatives  may  need  to  be  expanded  and  emphasized. 


DOD  must  work  with  contractors  to  develop: 

•  Conunon  awareness  of  DOD  business  past  performance  vision/strategy 

•  Shared  understanding  of  current  reality/leverage  points 

•  Align  actions  for  redesigning  processes  and  implementing  resource,  technology, 
and  organizational  features 

•  Collaborative  review  of  progress  throughout  the  cycle 


Long-term  contracts  should  be  emphasized  to: 

•  Provide  contractors  with  the  confidence  to  do  necessary  long-term  planning  and 
increase  commitment. 

•  Provide  contractors  with  tangible  evidence  that  you  are  serious  about 
partnership. 

•  Reduce  cost  by  lengthening  the  period  that  contractors  have  to  recover  capital 
investments. 

•  Reduce  administrative  costs  of  aimual  contract  award. 
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On-going  process  evaluation  programs  should  be  considered.  DCMC's  Risk 
Assessment  Model  and  PROCAS  programs  and  planned  initiatives  like  JACG-CPARS 
Supplier  Assessment  are  similar  to  industry  programs. 

•  DCMC  uses  RAM  and  PROCAS  programs  to  evaluate  key  contractors  and  to 
improve  tbeir  processes. 

•  Designed  fOT  contract  management  not  to  support  source  selection 

•  Do  not  apply  to  most  DOD  suppliers 

•  JACG-CPARS  focus  on  contractor  capability  to  perform  future  contracts  based 
on  performance  risk  and  assessment  of  key  processes. 

Process  evaluation  adds  an  important  dimension  to  judging  future  performance. 
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APPENDIX  A 


Study  Methodology 


The  principal  steps  in  our  methodology  for  this  study  were: 

•  Research  •  Model  Program  Development 

•  Review  •  Functional  Requirements 

•  Interviews  Development 

•  Analysis  •  Business  Case  Analysis 

•  Benchmarking 


A  description  of  the  major  activities  that  occurred  as  these  steps  were  applied  and  the 
results  of  these  efforts  are  included  in  Section  n  of  the  report.  A  brief  overview  for  each 
step  follows. 


Research 

Government,  industiy,  and  Arthur  D.  Little  research  resources  were  used  to  provide  a 
starting  point  for  the  study.  Our  research  at  the  start  of  the  study  was  focused  in  three 
areas: 

•  History  of  past  performance  information  systems  in  DOD.  We  examined  the 
history  of  other  past  performance  efforts  in  DOD  since  the  1960s.  The  collection 
and  use  of  past  performance  information  in  source  selection  decisions  is  not  new- 
numerous  approaches  have  been  tried,  and  this  historical  perspective  has  proven 
useful  in  guiding  the  study  as  well  as  developing  our  recommendations. 

•  Policy,  Legislation,  Regulations,  and  Guidance. 

•  The  principal  documents  reviewed  included: 

•  OFPP’s  Policy  Letter  92-5,  Past  Performance  Information,  December  30, 1992 

•  Section  1091  of  the  Federal  Acquisition  Streamlining  Act  of  1994  (FAS A) 

•  Federal  Register,  March  31, 1995,  Federal  Acquisition  Regulation,  Past 
Performance  Information,  Final  Rule 

•  OFPP’s  “A  Guide  to  Best  Practices  for  Past  Performance,”  Interim  Edition, 
May  1995 

•  Proposed  amendment  to  Defense  Federal  Acquisition  Regulation  Supplement 
(DFARS) 

•  DUSD  (AR)  and  Past  Performance  Coordinating  Council  information 
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Arthur  D.  Little’s  Industry  Supply  Chain  Management  Database.  Arthur  D.  Little 
maintains  a  database  of  Industry  Supply  Chain  Management  case  histories  which 
was  reviewed. 


Reviews  of  Past  Performance  Processes  and  Systems 

A  comprehensive  review  process  was  used  to  identify  and  categorize  existing  past 
performance  information  systems  in  the  federal  sector.  The  approach  we  followed  was  to 
first  identify  all  systems  and  processes  that  contained,  or  in  some  way  dealt  with,  past 
performance  information.  They  are  as  follows: 


Acronym 

System/Process  Name 

Owner 

1 

ABVM 

Automated  Best  Value  Model 

DLA 

2 

ACASS 

A&E  Contract  Administration  Support  System 

COE 

3 

ACPS 

Automated  Contract  Preparation  System 

Air  Force 

4 

ACTS 

Automated  Configuration  Tracking  System 

DCMC 

5 

AMIS 

Acquisition  Management  Information  System 

Air  Force 

6 

BCAS 

Base  Contracting  Automation  System 

Air  Force 

7 

BRP 

Blue  Ribbon  Program 

AUDOD 

8 

assR 

Cost/Schedule  Status  Reports 

DOD 

9 

CCASS 

Construction  Contract  Appraisal  Support  System 

COE 

10 

CCSS 

Commodity  Command  Standard  System 

Army 

11 

CDCS 

Customer  Depot  Complaint  System 

DLA 

12 

CIS 

Contractor  Information  System 

Army 

13 

CIS 

Contractor  Information  Service 

DCMC 

14 

CPARS 

Ccmtract  Performance  Assessment  Reporting  System 

Air  Force 

15 

CPR 

Cost  Performance  Reports 

DOD 

16 

CPS 

Contractor  Profile  System 

DCMC 

17 

DPACS 

DLA  Pre-award  Contracting  System 

DLA 

18 

GIDEP  Alerts 

Government  Industry  Data  Exchange  Program  Alcrts/Safe 

Alerts 

DOD 

19 

J041 

Acquisition  and  Due  In  System 

Air  Force 

20 

JACG-IPT 

Joint  Aeronautical  Commanders  Croup  Bitegratcd  Product  Team 
(study  covers  contractor  past  performance  and  supplier  rating) 

Joint  Service/  DLA 

21 

MIR 

Material  Inspection  Records 

Navy 

22 

MOCAS 

Mechanization  of  Contract  Administrative  Services 

DCMC 

23 

PADDS 

Procurement  Automated  Data  and  Document  System 

Army 

24 

PASS 

Pre-award  Survey  System 

DCMC 

25 

PDREP 

Product  Deficiency  Reporting  and  Evaluation  Program 

Navy 

26 

Product  Quality  Deficiency  Reports 

DCMC 

27 

PRAG 

Performance  Risk  Assessment  Groups 

Army/AF 

28 

prcx:as 

Process  Oriented  Contract  Administration  Services 

DCMC 

29 

QPL 

Qualified  Parts  List 

Navy 

30 

RAM 

Risk  Assessment  Model 

DCMC 

31 

RYG 

Red  Yellow  Green 

Navy 

32 

SAACS 

Standard  Army  Automated  Contracting  System 

Array 

33 

SALT 

System  Analysis  and  Lab  Testing 

DLA 

34 

SAMMS 

Standard  Automated  Material  Management  System 

DLA 

35 

VRS 

Vendor  Rating  System 

Air  Force 

Figure  A-1.1:  Systems  and  Processes  Related  to  Past  Performance 
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In  the  next  step,  we  used  the  definition  of  past  performance  information  to  focus  on  the 
more  relevant  systems  and  processes.  According  to  the  OFPP  PoUcy,  past  petformance 
information  regarding  a  contractors  actions  under  previously  awarded  contracts  is 
relevant  information.  Past  performance  information  includes  the  contractor’s; 

•  Record  of  conforming  to  specifications  and  to  standards  of  good  workmanship 

•  The  contractor  s  record  of  containing  and  forecasting  costs  on  any  previously 
performed  cost  reimbursable  contracts 

•  Adherence  to  contract  schedules,  including  the  administrative  aspects  of 
performance 

•  History  for  reasonable  and  cooperative  behavior  and  commitment  to  customer 
satisfaction 

•  Business-like  concern  for  the  interest  of  the  customer 


The  next  step  involved  further  screening  of  the  systems  and  processes  using  the  definition 
for  past  performance  information  systems.  This  definition  was  derived  from  guidance 
and  direction  contained  in  OFPP  PoUcy  Letter  No.  92.5,  FAR  changes  (FAC  90-26),  and 
OFPP  A  Guide  to  Best  Practices  for  Past  Performance.”  The  definition  used  for  past 
petformance  information  systems  is  as  follows: 

•  Information  is  collected,  vaUdated  and  filled  for  the  specific  purpose  of  supporting 
future  source  selection  decisions 

•  Opportunity  is  provided  for  review,  comment  and  rebuttal  of  the  information  by 
the  contractor 

•  Provisions  are  estabUshed  to  resolve  disputes  between  the  contractor  and  the 
government 

•  Information  is  subject  to  the  same  controls  and  safeguards  as  other  information 
used  in  source  selection  decisions 

•  System  is  in  operation  and  currently  supporting  source  selection  decisions 

Interviews 

We  conducted  interviews  to  obtain  current  information  on  existing  past  performance 
processes  and  systems.  The  chart  below  illustrates  the  approach  we  used  to  gather  data 
on  existing  past  performance  process  and  systems. 
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Arthur  D.  Little  Supply 
Chain  Management 
Commercial  Database 


ODUSD(AR),  DLA, 
SERVICES,  PPCC 
Information 


Task  2  Criteria 


FASA,  FAR,  DFAR 
Guidance 


Information  needed  for 
rask4A  &6 


Industry 

•  DOD>related 

•  No/Minimum  DOD 
involvement 


DOD 

•  OSD,  Services, 
Agencies 


Other  Government 
•NASA.  DOE, GSA 


Data  to  construct  the  interview  guides  was  obtained  fix)ni:  Arthur  D.  Little  supply  chain 
management  databases;  information  provided  by  DUSD  (AR);  the  evaluation  criteria 
contained  in  the  statement  of  work  for  the  study  provided  above;  data  from  the  relevant 
Federal  Acquisition  Streamlining  Act,  Federal  Acquisition  Regulation,  and  Office  of 
Federal  Procurement  Policy  documents;  and  questions  for  information  we  anticipated 
would  be  needed  later  in  the  study. 


The  interview  guides  and  their  application  provided  a  consistent,  structured  approach  to 
data  collection.  As  indicated  in  the  right-hand  side  of  the  above  chart,  interviews  were 
conducted  in  three  major  sectors~DOD,  including  OSD,  the  DOD  component;  industry, 
including  both  defense  and  coimnercial  contractors;  and  non-DOD  Government  agencies, 
including  the  National  Aeronautics  and  Space  Administration,  Department  of  Energy, 
Department  of  Commerce,  General  Services  Administration,  and  Department  of 
Transportation. 


An  important  outcome  of  this  approach  was  the  broad  industry  and  government  response 
we  were  able  to  obtain.  Such  response  was  significant  in  that  it  assured  broad 
representation  and  helped  mitigate  potential  bias.  A  wide  range  of  DOD  organizations 
was  contacted  for  information  relevant  to  this  study.  Interviews  were  conducted  with 
representatives  of;  Defense  Logistics  Agency  (DLA)  at  Headquarters  and  at  the  Defense 
General  Supply  Center;  Defense  Contract  Management  Command;  Assistant  Secretary  of 
the  Army/Research,  Development,  and  Acquisition;  Army  Material  Command;  Army 
Corps  of  Engineers;  Secretary  of  the  Air  Force/Acquisition  Contracting;  Air  Force 
Material  Command/Procurement  and  Wright  Laboratories;  Assistant  Secretary  of  the 
Navy/Research  Development  and  Acquisition;  Naval  Air  Systems  Command;  and  Naval 
Material  Quality  Assurance  Office.  Non-DOD  Federal  Agencies  were  also  contacted. 
GSA  in  particular  provided  significant  coverage  in  terms  of  contracting  experience  since 
GSA  manages  many  contracts  that  serve  other  federal  government  agencies.  NASA 
provided  a  technology  perspective  and  DOE  the  perspective  of  complexity.  Such  wide 
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organizational  representation  also  provided  a  diversity  of  experience  in  terms  of  the 
nature  of  the  products  and  services  that  were  acquired. 


It  is  important  to  note  that  in  conducting  the  interviews,  information  was  gathered  not 
only  from  users  of  past  performance  systems  and  processes,  but  also  from  the  managers 
and  owners  of  such  systems  and  processes.  This  approach  provided  assessments  and 
ideas  from  many  individuals  representing  the  different  points  of  view  in  the  process. 
Together,  the  DOD  and  non-DOD  sources  of  information  provided  a  relatively  large 
experience  base  to  draw  upon.  This  base  was  a  particular  strength  of  this  study.  The 
interview  guides  are  included  in  Appendix  A. 


Analysis 

Analyses  were  conducted  for  each  of  the  existing  systems  to  determine  the  success  of 
each  system  in  meeting  its  past  p^ormance  information  system  objectives.  Odier 
analyses  examined  the  extent  of  coverage  provided  by  the  existing  systems  relative  to 
contract  dollar  value,  product  or  service  areas,  and  the  evaluation  factors  cited  in  the 
OFPP  guide  on  Contractor  Past  Performance. 


In  addition,  each  of  the  systems  and  processes  was  compared  to  the  evaluation  factors 
contained  in  the  Statement  of  Work.  These  factors  covered  the  following: 

•  Data  System  Design— centralized  or  non-centralized 

•  Kinds  of  data  used— government,  private 

•  Integrity  of  data— identity  of  sources 

•  Accuracy 

•  Currency 

•  Remedial  Process  by  Contractors 

•  Availability  of  Information  for  Source  Selection 

•  Confidentiality 

•  Sub-contractor  Involvement 

•  Maintaining  Identity  of  Contractors  That  are  Acquired 

•  Fairness 

•  Due  Process 

•  Lack  of  Past  Performance 

•  Threshold  of  Applicability 

•  Capability  of  Attribution 

•  Penalty 
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Each  of  the  DOD  components  have  initiatives  underway  which  aim  at  expanding  the  past 
performance  information  available  for  use  in  contractor  selection  decisions.  Planned  past 
performance  information  sources  were  identified  in  the  study  along  with  the  conceptual 
approach  that  will  be  applied  and  the  depth  of  coverage. 


Systems  used  by  other  government  agencies  were  also  analyzed,  focusing  on  the 
following  four  areas; 

•  Published  Policies 

•  Rating  System 

•  Databases 

•  Known  problems  with  existing  approach  to  past  performance  evaluation 

Benchmarking 

The  benchmarking  phase  of  the  study  was  developed  through  on-site  visits  and  the 
review  of  information  in  the  Arthur  D.  Little  Supply  Chain  Management  practice 
database  and  secondary  research.  For  benchmarking  purposes  we  interviewed  companies 
which  are  considered  to  be  best  in  class  in  terms  of  supplier  past  performance  evaluation. 
These  firms  included; 

•  Allen-Bradley 

•  Baxter  Health  Care 

•  Black  &  Decker 

•  Boeing  Defense  and  Space  Group 

•  Fisher  Scientific 

•  Ford  Motor  Co.  (by  telephone) 

•  McCormick  &  Co. 

•  McDonnell  Douglas/C-17 

•  Mobile  Corporation 

•  National  Semiconductor 

•  Rockwell  North  American 

•  U.S.  Postal  Service 

•  W.W.  Grainger 


We  compiled  the  results  into  a  series  of  “best  practices”  that  were  briefed  to  DUSD(AR) 
on  January  24, 1996. 


Model  Program  Development 

The  Contractor  Evaluation  Program  is  the  model  program  we  developed  to  assist 
DUSD(AR)  in  past  performance  policy  implementation.  We  approached  the 
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development  of  the  Contractor  Evaluation  Program  by  refining  our  benchmarking  results 
and  by  conducting  workshops  for  DOD  officials  who  were  involved  in  acquisition  reform 
initiatives  and  who  represented  the  functional  areas  that  were  affected  in  some  way  by 
contractor  past  performance  processes  and  systems. 


In  so  doing,  several  workshops  were  held  with  the  following  representatives  from  the 
Acquisition  Reform  Senior  Steering  Group  (ARSSG)  and  the  Past  Performance 
Coordinating  Committee  (PPCC): 

•  Major  Programs  (API) 

•  Logistics 

•  Economic  Security 

•  Systems  Engineering 

•  Quality 

•  Inspector  General 

•  Procurement 

•  General  Counsel 

•  Defense  Contract  Management  Command  (DCMC) 

•  Defense  Contract  Audit  Agency 


Workshops  were  built  around  four  inter-related  modules,  listed  below: 

•  Review  background  information  (address  new  policies;  government  and  industry 
programs) 

•  Develop  working  definition  of  contractor  past  performance 

•  Assess  selected  contractor  past  performance  evaluation  practices 

•  Develop  a  working  process  for  contractor  past  performance 


Workshop  participants  were  introduced  to  the  goals,  objectives,  and  desired  outcome  of 
the  study.  In  order  to  provide  a  baseline  for  each  workshop,  information  was  provided  to 
participants  on  DOD  5000  and  FAR/DFARS,  as  well  as  on  common  elements  associated 
with  a  contractor  evaluation  program.  In  addition,  industry  supplier  evaluation  programs 
and  lessons  learned  from  industry  were  shared  with  participants. 


During  the  workshops,  maximum  opportunity  was  provided  for  participants  to  share  their 
perspectives  on  past  performance  evaluation.  Participants  also  addressed  questions 
concerning  a  DOD  contractor  vision  and  implications  of  anticipated  changes  for  the 
acquisition  community. 
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Functional  Requirements  Development 

The  Contractor  Evaluation  Program  was  analyzed  to  develop  a  functional  requirements 
document  which  was  provided  to  DUSD(AR)  as  a  separate  deliverable. 

Business  Case  Analysis 

The  business  case  analysis  focused  on  an  assessment  of  the  alternate  approaches  for 
implementing  Past  Performance  policy.  Information  was  addressed  in  three  areas: 

•  The  current  processes  and  systems  that  deal  with  contractor  past  performance; 

•  The  recent  changes  in  the  processes  and  systems  that  are  directed  by  the  FAR  and 
the  proposed  changes  to  the  DFARS;  and 

•  A  proposed  q)proach  for  dealing  with  contractor  past  performance  issues,  referred 
to  as  the  Contractor  Evaluation  Program 

The  analysis  covers  process  mapping,  automated  data  information  system  analysis,  and  a 
comparison  of  the  differences  between  the  proposed  Contractor  Evaluation  Program 
model  and  the  FAR/DFARS  approach. 
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Past  Performance  Process  Analysis 

Three  contractor  past  performance  systems  were  reviewed,  RedA^ellow/Green 
(RYG),  the  Automated  Best  Value  Method  (ABVM),  and  the  Architect-engineer  Contract 
Administration  Support  System  (ACASS).  Process  flows  were  developed  for  each  system 
using  a  standard  approach  that  was  briefed  on  24  January  1996. 


.^though  all  three  systems  are  geared  towards  very  diff^ent  types  of  purchases, 
the  functions  required  to  operate  all  three  systems  is  nearly  identical.  All  three  systems 
require  the  collection  of  data  from  various  sources.  All  three  systems  use  both  manual  and 
electronic  inputs.  Both  ABVM  and  RYG  use  electronic  inputs  from  other  systems, 
whereas  ACASS  uses  electronic  performance  evaluations  from  the  U.S.  Army  Corps  of 
Engineers  Engineering  Division.  All  three  have  a  process  in  place  to  resolve  contractor 
challenges.  Both  the  RYG  and  ACASS  systems  include  the  negative  ratings  in  the 
contractor  information  that  is  used  by  buyers  with  an  annotation  that  the  rating  is  being 
challenged.  ABVM  does  not  include  ratings  that  are  being  challenged  in  the  contractor 
information  that  is  being  disserrunated.  The  RYG  system  tries  to  provide  an  automated 
adjustment  to  compensate  for  the  higher  risk  of  a  contractor  with  a  negative  rating  to  help 
buyers  in  their  selection  choice.  ABVM  and  ACASS  provide  the  ratings,  and  require  the 
buyers  to  use  their  judgment  to  make  the  Best  Value  procurement. 

Each  of  the  activities  was  desegregated  and  the  processes  that  occur  for  each  of 
the  above  major  activities  is  documented  in  greater  detail  in  the  following  pages. 
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Red  AT  ellow/Green  System  Process 

The  RedATellow/Green  system  is  a  Navy  system  that  provides  historical 
performance  data  to  buyers.  The  system  is  currently  being  hosted  on  a  mainframe 
computer  at  the  Naval  Sea  Systems  Command  (NAVSEA)  Detachment  Naval  Material 
Quality  Assessment  Office  (NMQAO)  at  Portsmouth  New  Hampshire.  The  Product 
Deficiency  Reporting  and  Evaluation  (PDREP)  system  collects  data  from  a  variety  of 
sources.  RYG  classifications  are  automatically  assigned  based  on  contractor  performance 
data  collected.  PDREP  updates  the  RYG  tables.  System  Operators  post  vendor 
classifications  on  a  Bulletin  Board  System,  and  buyers  download  this  data  via  modem. 
Buyers  download  the  most  recent  classifications  onto  their  PC  where  the  RYG  software 
resides. 

Buyers  review  the  RYG  tables,  and  for  those  vendors  with  Yellow  or  Red 
Classifications,  they  must  decide  to  use  the  TEAs  or  the  Greatest  Value/Best  Buy  method. 
If  they  decide  to  use  TEAs,  almost  all  use  the  standard  TEA  which  is  automatically 
calculated  by  PDREP.  Once  all  vendor  data  is  assembled,  the  buyer  compares  the 
adjusted  prices  and  makes  an  award. 

Red/Yeliow/Green  Major  Processes 


Each  of  the  major  process  activities  has  been  disaggregated  into  subcomponent  processes: 
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A.  Collect,  Generate,  Calculate,  and  Validate  Performance  Information 

Activity  A  of  the  major  ABVM  process  has  been  disaggregated  here.  This  activity 
pertains  to  the  collection  of  contractor  performance  data;  verifying  that  the  data  is  correct; 
automatically  generating  contractor  classifications  -  either  a  Red,  a  Yellow,  or  a  Green; 
satisfying  contractor  challenges;  and  releasing  the  classifications  for  use  by  buyers. 


Red/Yellow/Green 

Collect,  Generate,  Calculate  &  Validate 
Performance  Information 


A,9 

Government 

Review 


A.6 

_ 5 

A.10 

Yes 

A.12 

Correct  Data 

1 

uoniractor 

> - > 

Resolve 

Challenge 

1 

A.l  Elex  Record  -  Electronic  records  to  be  input  into  PDREP.  This  activity  is  included 
here  as  a  starting  place  for  this  process. 

A.2  Receive  Elex  Data  -  Maintain  interactive  input  programs,  batch  import  programs,  and 
error  checking  algorithms.  This  activity  is  explored  in  D.3  “Maintain  Interfaces,”  but 
included  here  because  it  is  a  part  of  this  process.  On  the  average  for  any  given  month; 

1.  7,500  individual  reports  are  electronically  transferred  for  import  into  PDREP. 

2.  These  transfers  are  made  via  Bulletin  Board,  various  PDREP  applications,  hard 
diskettes,  and  CEDES. 

3.  Two  full-time  programmers  are  required  to  maintaining  these  data  links. 

A.3  Hard  Record  -  paper  record  input  into  PDREP.  This  activity  is  included  here  as  the 
other  starting  place  for  this  process. 
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A.4  Receive  Hard  Copy  Data  -  Receive,  review,  validate  and  correct  paper  reports.  lOn 
the  average  for  any  givai  month: 

1.  750  individual  hard  copy  reports  are  received  for  input  into  PDREP. 

2.  It  takes  data  input  personnel  from  one  to  twenty  days  calendar  days  and  forty 
labor  hours  to  validate  the  hard  copy  data. 

3.  Once  data  errors  are  discovered,  it  can  take  one  to  twenty  days  calendar  days 
and  forty  hours  of  labor  to  correct  the  hard  copy  data? 

A.5  Valid  Electronic  Data  -  review  data  and  determine  its  validity.  On  the  average  for  any 
given  month: 

1.  It  takes  programmers  from  one  to  twenty  days  calendar  days  and  eight  labor 
hours  to  validate  the  electronic  data. 

A.6  Correct  Data  -  re-import  data,  reject  &  return  to  data  source,  manually  correct  data. 
On  the  average  for  any  given  month: 

1 .  It  takes  data  processing  personnel  from  one  to  twenty  days  calendar  days  and 
eight  labor  hours  to  correct  the  electronic  data. 

A.7  Enter  Data  -  import  electronic  data  and  key  in  hardcopy  data  into  PDREP  database. 
On  the  average  for  any  given  month: 

1 .  It  takes  data  processing  personnel  from  one  to  twenty  days  calendar  days  and 
ten  labor  hours  to  maintain  the  import  programs  and  import  electronic  data. 

2.  It  takes  data  entry  personnel  forty  mandays  to  key  in  the  data  from  hard  copy 
reports. 
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A.8  Calculate  aassifications  -  assign  RYG  classiJBcation  to  contractors.  These 
calculations  are  provided  automatically  by  the  PDREP  software  to  the  RYG  system. 

A.9  Government  Review  -  government  personnel  review  RYG  classifications  for  accuracy 
&  concurrence.  On  the  average  for  any  given  month: 

1.  Government  personnel  take  1-7  workdays  and  56  labor  hours  to  review  RYG 
records. 

A.  10  Contractor  Review  -  provide  classification  rqiort  to  contractors.  On  the  average  for 
any  given  month: 

1.  5000  reports  are  provided  to  contractors. 

2.  It  takes  coordinator  personnel  from  one  to  seven  days  calendar  days  and  4 
labor  hours  to  provide  these  reports  to  contractors. 

A.l  1  Challenge  -  does  the  contractor  challenge  the  rating? 

1 .  The  contractor  has  twenty  work  days  to  issue  a  challenge. 


A.  12  Resolve  challenge  -  work  with  contractor  and  users  to  resolve  any  rating  challenges. 
On  the  average  for  any  given  month: 

1 .  Fi%  challenges  are  received  and  resolved. 

2.  This  takes  program  managers  from  1-20  days  and  15  mandays  of  labor  to 
complete. 
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A.  13  Modify,  Authorize  Use  -  Update  PDREP  classifications  as  a  result  of  challenge 
resolution  and  authorize  use  of  the  data  for  contractor  evaluation  purposes.  On  the 
average  for  any  given  month: 

1 .  10  records  are  modified  as  a  result  of  challenge  resolution. 

2.  Coordinators  take  form  1-20  days  and  four  labors  hours  to  make  the  necessary 
changes. 
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B.  File,  Store,  Maintain,  and  Provide  Access  to  Performance  Information 

Activity  B  pertains  to  storing  the  contractor  information  electronically  such  that 
buyers  can  access  contractor  historical  performance  information.  Buyers  can  access  RYG 
data  directly  by  dialing-up  to  the  database  via  modem.  If  buyers  don’t  have  access  to  a 
modem,  their  network  administrator  dials-up,  download  the  data  to  a  bulletin  board  on  the 
local  area  network  (LAN)  where  buyers  can  then  access  the  information. 


Red/ Yel  I  o  w/G  reen 
File,  Store,  Maintain  Data 


B.l  Download  RYG  info  into  Bulletin  Board  System  -  extract  RYG  classification  info 
from  PDREP  database  into  Bulletin  Board  System.  This  activity  is  explored  in  D.3 
“Maintain  Interfaces.” 

B.2  Buyer  have  Modem  -  determine  whether  the  buyer  has  a  modem. 

B.3  Buyer  dial-up  and  access  data  -  if  the  buyer  has  a  modem,  RYG  classifications  can  be 
electronically  downloaded  from  the  RYG  Bulletin  Board  System  and  used  in  contractor 
evaluation.  On  the  average  for  any  given  month: 

1 .  One  buyer  for  each  buying  activity  (22)  access  the  RYG  classifications  via 
modem.  This  occurs  once  a  month,  and  the  download  takes  twenty  minutes. 
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Red/Yellow/Green 
File,  Store,  Maintain  Data 


B.4  Network  Admin  Dial-up  &  download  data  -  if  the  buyer  does  not  have  a  modem,  the 
local  area  network  (LAN)  administrator  will  dial-up  and  download  the  RYG  classification 
information.  On  the  average  for  any  given  month: 

1.  The  LAN  administrator  for  each  buying  activity  (22)  downloads  the  RYG  data 
onto  RYG  bulletin  boards  on  their  LANs.  This  occurs  once  a  month,  and  the 
download  takes  twenty  minutes. 

2.  TTie  LAN  administrators  each  take  four  hours  to  maintain  the  bulletin  board  on 
their  LAN. 


B.5  Buyer  access  data  via  LAN  -  buyers  access  RYG  classifications  downloaded  by  their 
LAN  administrators  via  their  LAN.  On  the  average  for  any  given  month: 

1 .  Two  thousand  buyers  access  RYG  classifications  via  their  LANs  on  a  daily 
basis,  taking  only  seconds. 

B.6  Buyer  Use  Data  -  buyers  use  this  data  to  make  source  selections.  This  is  discussed  in 
Activity  C.  Retrieve,  Gather,  and  Use  Contractor  Past  Performance  Information. 
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C.  Retrieve,  Gather  and  Use  Contractor  Past  Performance  Information 

Activity  C  pertains  to  the  use  of  contractor  performance  information  by  buyers  as 
part  of  the  buying  process.  If  a  vendor  has  a  yellow  or  red  classification,  the  buyer  must 
use  a  Technical  Evaluation  Adjustment  (TEA).  A  Standard  TEA  is  automatically 
calculated  by  the  RYG  software.  A  site  with  a  Quality  Assurance  (QA)  Depaitinent  can 
develop  their  own  TEA  based  on  their  experience  with  the  contractor.  The  TEA  adjusts 
the  bid  for  risk  such  that  the  lowest  adjusted  bid  can  be  awarded  the  job. 


Retrieve,  Gather 
and  Use 
Contractor 
Past  Performance 
Information 


C.1  Retrieve  Classifications  -  buyer  will  retrieve  classifications  for  contractors  who  have 
provided  a  bid  from  the  current  RYG  classifications  loaded  onto  their  PCs  in  B.  “File, 
Store,  Maintain,  and  Provide  Access  to  Performance  Information.”  On  the  average  for  any 
given  month: 

1.  Buyers  process  ten  buys,  reviewing  3  contractors  for  each  buy. 

2.  Buyers  do  not  review  the  RYG  classifications  for  every  buy. 

C.2  Yellow  or  Red?  -  does  the  contractor  have  a  Red  or  Yellow  classification?  On  the 
average  for  any  given  month 

1.  Only  2%  of  Contractors  reviewed  have  a  Yellow  or  Red  Classification. 

C.3  Use  TEA  or  GV/BB?  -  the  buyer  decides  whether  to  use  the  TEA  factors  or  the 
Greatest  Value  /  Best  Buy  method.  The  GV/BB  is  more  appropriately  used  in  high  dollar 
value/large  purchases  where  differences  in  vendor’s  prices  are  so  large,  that  the  TEA 
comparisons  cannot  be  made. 
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C4  Use  Customized  TEA?  -  the  buyer  decides  whether  to  use  a  customized  TEA  or  the 
standard  TEA  provided  by  the  RYG  software. 

Even  though  approximately  500  buyers  have  access  to  QA  departments  and  could  develop 
customized  TEAs. 

C.5  Use  Standard  TEA  -  buyers  use  the  standard  TEAs  automatically  developed  in  the 
RYG  software  based  on  contractor  performance.  On  the  average  for  any  given  month: 

1 .  99%  of  all  TEAs  used  are  based  on  the  standard  values  produced 
automatically. 

C.6  Develop  Customized  TEA  -  for  those  buying  sites  equipped  with  quality  assurance 
departments,  on  the  average  for  any  given  month: 

1 .  Virtually  no  customized  TEAs  are  developed. 

C.7  Compare  Quotes  &  Select  Source  -  buyers  using  either  the  TEA  or  GV/BB  method 
will  compare  quotes  and  make  an  award.  On  the  average  for  any  given  month: 

I .  Four  hundred  contracts  are  awarded. 
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D.  Design,  Operate  and  Manage  the  Automated  Elements  of  the  Process 

Activity  D  pertains  to  the  computer  and  communication  system  management 
System  operators  maintain  the  databases,  update  documentation,  upgrade  SW  &  HW  as 
new  versions  are  released,  maintain  and  upgrade  user  interfaces,  and  ensure  that  import 
and  export  functions  operate  properly. 


Red/Y ellow/Green 

Design,  Operate  and  Manage  the  Automated 
Elements  of  the  Process 


D.l  Maintain  guides  -  guides,  procedures,  handbooks  for  the  RedA'ellow/Green  system 
must  be  kept  up  to  date  and  current,  documenting  any  changes  in  procedure, 
responsibility,  and  data  elements. 

1.  Documentation  for  the  RYG  system  is  updated  once  per  year,  requiring  ten 
mandays  of  labor. 

D.2  Manage  Database  -  Maintaining  the  RYG  database  requires  2  manyears  of  labor 
annually. 

D.3  Maintain  Interfaces  -  Maintaining  the  automated  interfaces  and  bulletin  boards 
requires  2  manyears  of  labor  annually. 

D.4  Analyze  Database  -  Analyzing  the  data,  data  elements,  normalizing  the  data,  etc. 
requires  2  manyears  of  labor  annually. 

D.5  Analyze/Publish  Statistics  -  requires  1  manyear  of  labor  annually. 

D.6  Upgrade/Enhance  Operation  -  upgrading  HW  &  SW  requires  2  manyears  of  labor 
annually. 
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RedA'  ellow/Green 

Design,  Operate  and  Manage  the  Automated 
Elements  of  the  Process 


D.7  RecVYellow/Green  System  -  this  encompasses  the  entire  system  that  provides  the 
RYG  classifications  and  their  use  (A,B,  &  C). 
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E.  Manage  and  Oversee  the  Total  Past  Performance  Information  Process 

Activity  E  refers  to  the  administration  and  overall  management  of  the 
RedA^ellow/Green  system.  Administrative  personnel  provide  information  about  the 
system,  provide  training,  oversee  system  operations,  and  perform  administrative  tasks. 

Red/Yellow/Green 

Manage  and  Oversee  the  Total  Past 

Performance  Information  Process 


E.1  Respond  to  Requests  -  it  requires  1  full  time  equivalent  employee  to  respond  to 
routine  and  special  requests  for  information. 

E.2  Manage  Data  -  it  requires  1  full  time  equivalent  employee  to  manage  the  information 
produced  for  and  by  the  RYG  system. 

E.3  Manage  System  Ops  -  it  requires  1  full  time  equivalent  employee  to  manage  the  day  to 
day  operational  activities. 

E.4  Admin  Functions  -  it  requires  one  fourth  of  a  full  time  equivalent  employee  to  take 
care  of  administrative  activities. 

E.5  Respond  to  Questions  -  it  requires  one  half  of  a  full  time  equivalent  employee  to 
respond  to  questions  about  the  system,  its  operation,  and  personnel. 

E.6  Provide  Training  -  it  requires  one  fourth  of  a  full  time  equivalent  employee  to  provide 
training  to  RYG  users. 
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Red/Yellow/Green 

Manage  and  Oversee  the  Total  Past 

Performance  information  Process 


E.7  Red/Yellow/Green  Process  -  this  encompasses  the  entire  system  that  provides  the 
RYG  classifications,  their  use,  and  tiie  information  systems  used  (A,B,C,  &  D). 
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Automated  Best  Value  Model  System  Process 


The  Automated  Best  Value  Model  (ABVM)  System  is  a  Defense  Logistics  Agency 
(DLA)  system  that  provides  historical  performance  data  to  buyers.  The  system  is 
currently  being  hosted  on  a  mainframe  computer  at  the  Information  Processing  Center  at 
Columbus  Ohio.  Contractor  rating  information  is  transferred  to  the  Defense  Electronics 
Supply  Center  (DESC),  the  Defense  Supply  Center  Columbus  (DSCC),  the  Defense 
Supply  Center  Richmond,  and  the  Defense  Industrial  Supply  Center  via  a  Wide  Area 
Network  (WAN).  Access  by  buyers  is  provided  via  a  local  area  network  (LAN). 

ABVM  automatically  calculates  a  rating  using  inputs  such  as  product  quality 
nonconformances,  packaging  nonconformances,  laboratory  test  results,  delinquencies,  and 
order  rejections  where  the  company  has  demonstrated  an  intent  to  perform.  Buyers  must 
trade-off  a  price  for  past  performance  when  the  contractor  with  the  highest  ABVM  score 
does  not  have  the  lowest  price.  Other  factors  that  buyers  consider  are  item  designation  as 
weapons  system  or  personnel  support  item,  inventory  supply  status  and  required  delivery 
schedule,  limited  sources  of  supply  and  industrial  base  concerns,  dollar  difference  between 
low  technically  acceptable  offeror  and  a  higher  priced,  higher  scored  offeror,  and  the 
presence  of  new  offerors. 


Automated  Best  Value  Model  System 
Major  Processes 


Each  of  the  major  process  activities  has  been  disaggregated  into  subcomponent  processes: 
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A.  Collect,  Generate,  Calculate,  and  Validate  Performance  Information 

Activity  A  of  the  major  AB  VM  process  has  been  disaggregated  here.  This  activity 
pertains  to  the  collection  of  contractor  performance  data;  verifying  that  the  data  is  correct; 
automatically  generating  contractor  ratings;  satisfying  contractor  challenges;  and  releasing 
the  classifications  for  use  by  buyers. 


A.  1  Elex  Record  -  Electronic  records  to  be  input  into  AB  VM  database.  This  activity  is 
included  here  as  a  starting  place  for  this  process. 

A.2  Receive  Elex  Data  -  Maintain  interactive  input  programs,  batch  import  programs,  and 
error  checking  algorithms.  This  activity  is  explored  in  D.3  “Maintain  Interfaces,”  but 
included  here  because  it  is  a  part  of  this  process. 

A.3  Hard  Record  -  paper  record  input  into  ABVM  database.  This  activity  is  included  here 
as  the  other  starting  place  for  this  process. 

A.4  Receive  Hard  Copy  Data  -  Receive,  review,  validate  and  correct  paper  reports. 

A.5  Valid  Electronic  Data  -  review  data  and  determine  its  validity. 

A.6  Correct  Data  -  reimport  data,  reject  &  return  to  data  source,  manually  correct  data. 
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A.7  Enter  Data  -  import  electronic  data  and  key  in  hardcopy  data  into  ABVM  database. 


A.8  Calculate  Classifications  -  assign  ABVM  classification  to  contractors.  These 
calculations  are  provided  automatically  by  the  ABVM  database  software  to  the  ABVM 
system. 


A.9  Contractor  Review  -  computed  ratings  and  detailed  negative  performance  data  via  a 
PC  based  bulletin  board  system. 

A.  10  Challenge  -  does  the  contractor  challenge  the  rating? 

A.l  1  Resolve  challenge  -  work  with  contractor  and  users  to  resolve  any  rating  challenges. 

A.  12  Modify,  Authorize  Use  -  Update  ABVM  database  ratings  as  a  result  of  challenge 
resolution  and  authorize  use  of  the  data  for  contractor  evaluation  purposes. 
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B.  File,  Store,  Maintain,  and  Provide  Access  to  Performance  Information 

Activity  B  pertains  to  storing  the  contractor  information  electronically  such  that 
buyers  can  access  contractor  historical  performance  information.  ABVM  data  is 
transferred  using  a  file  transfer  protocol  (FTP)  via  a  Wide  Area  Network  (WAN)  to  each 
buyer’s  site  and  is  accessed  by  the  buyers  via  their  local  area  network  (LAN), 


Automated  Best  Value  Model 
File,  Store,  Maintain  and 
Provide  Access  to 
Performance  Information 

■*r 

B.2 

Buyer  Access  Data 
via  LAN 


B.1 


iFTP  Rating  Info  to 
I  Supply  Centers  viaj 
WAN 


B.l  FTP  Rating  Info  to  Supply  Centers  via  WAN  -  ABVM  system  administrators 
electronically  transfer  ABVM  Rating  information  to  Supply  Center  networic  administrators 
who  then  make  this  information  available  to  buyers  on  the  local  area  network  (LAN). 

This  activity  is  explored  in  D.3  “Maintain  Interfaces.” 

B.2  Buyer  Access  Data  via  LAN  -  LAN  administrators  download  the  FTP  transfer  of 
ABVM  information  into  the  local  installation  of  DLA  PreAward  Contracting  System 
(DPACS).  Buyers  access  contractor  rating  information  through  DPACS  when  making 
buys  on  their  local  area  network. 
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C.  Retrieve,  Gather  and  Use  Contractor  Past  Performance  Information 

Activity  C  pertains  to  the  use  of  contractor  performance  information  by  buyers  as 
part  o  t  e  buying  process.  As  the  name  of  the  system  suggests,  buyers  determine  the 
i^st  value  by  weighing  cost  and  risk.  If  a  contractor  submits  the  lowest  bid  and  the 
contractor  also  conveys  the  lowest  associated  risk,  the  buyer  simply  makes  an  award  to 
that  contractor.  If  the  buyer  makes  an  award  to  the  contractor  who  represents  the  overall 
best  value,  but  did  not  submit  the  lowest  bid,  the  buyer  must  write  an  award  justification. 


Past  Performance 
Information 


Cl  Retrieve  Vendor  Ratings  via  LAN  -  buyers  wiU  retrieve  contractor  ratings  when 
making  a  buy  via  DPACS  which  is  accessed  on  their  LAN. 

C.2  Lowest  Risk  and  Price?  -  does  any  bidding  contractor  provide  the  lowest  risk  based 
on  the  AB  VM  rating  and  offer  the  lowest  price?  If  one  does,  the  buyer  may  make  the 

award,  if  no  one  bidder  offers  the  lowest  risk  and  price,  the  buyer  must  compare  all  the 
bids. 

C.3  Compare  Quotes  and  Select  Best  Value  Source  -  the  buyer  makes  a  vendor  selection 
based  on  the  price  of  the  bid,  and  the  risk  associated  with  the  vendor  based  on  past 
performance  as  reflecting  the  assigned  vendor  rating. 

C.4  Wnte  Justification  -  the  buyer  must  write  a  justification  of  source  selections  whenever 
the  lowest  bidder  is  not  selected. 

C.5  Make  Award  -  the  buyer  issues  an  award  to  the  selected  vendor. 
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D.  Design,  Operate  and  Manage  the  Automated  Elements  of  the  Process 

Activity  D  pertains  to  the  computer  and  communication  system  management. 
System  operators  maintain  the  databases,  update  documentation,  upgrade  SW  &  HW  as 
new  versions  are  released,  maintain  and  upgrade  user  interfaces,  and  ensure  that  import 
and  export  functions  operate  properly. 

Automated  Best  Value  Model  System 
Design,  Operate  and  Manage  the  Automated 
Elements  of  the  Process 


D.l  Maintain  guides  -  guides,  procedures,  handbooks  for  the  Automated  Best  Value 
Model  system  must  be  kept  up  to  date  and  current,  documenting  any  changes  in 
procedure,  responsibility,  and  data  elements. 

D.2  Manage  Database  -  system  operators  must  manage  the  database  to  ensure  that  is 
properly  updated  and  used. 

D.3  Maintain  Interfaces  -  system  administrators  must  maintain  the  automated  interfaces 
for  both  incoming  data  and  transfers  to  DPACS. 

D.4  Analyze  Database  -  system  operators  must  analyze  the  database  to  ensure  data 
elements  are  not  corrupted,  and  the  data  remains  normalized. 

D.5  Analyze/Publish  Statistics  -  operating  reports  and  statistics  are  kept  up  to  date  and  are 
published  periodically  and  on  an  adhoc  basis. 

D.6  Upgrade/Enhance  Operation  -  system  operators  must  upgrade  HW  &  SW  ensuring 
that  only  up  to  date  versions  of  both  Commercial-Off-The-Shelf  (COTS)  HW  &  SW  and 
any  internally  developed  code  is  being  used. 
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Automated  Best  Value  Model  System 
Design,  Operate  and  Manage  the  Automated 
Elements  of  the  Process 


D.7  Automated  Best  Value  Model  System  -  this  encompasses  the  entire  system  that 
provides  the  ABVM  classifications  and  their  use  (A,B,  &  C). 
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E.  Manage  and  Oversee  the  Total  Past  Performance  Information  Process 

Activity  E  refers  to  the  administration  and  overall  management  of  the  Automated 
Best  Value  Model  System.  Administrative  personnel  provide  information  about  the 
system,  provide  training,  oversee  system  operations,  and  perform  administrative  tasks. 


Automated  Best  Value  Model  System 
Manage  and  Oversee  the  Total  Past 
Performance  Information  Process 


E.  1  Respond  to  Requests  -  system  administrators  respond  to  routine  and  special  requests 
for  information. 

E.2  Manage  Data  -  system  administrators  manage  the  information  produced  for  and  by  the 
ABVM  systera 

E.3  Manage  System  Ops  -  system  administrators  manage  the  day-to-day  operational 
activities. 

E.4  Admin  Functions  -  system  administrators  perform  necessary  administrative  activities. 

E.5  Respond  to  Questions  -  system  administrators  respond  to  questions  about  the  system, 
its  operation,  and  personnel. 

E.6  Provide  Training  -  system  administrators  provide  training  to  ABVM  users. 
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Automated  Best  Value  Model  System 
Manage  and  Oversee  the  Total  Past 
Performance  Information  Process 


E.7  Automated  Best  Value  Model  Process  -  this  encompasses  the  entire  system  that 
provides  the  ABVM  classifications,  their  use,  and  the  information  systems  used  (A,B,C,  & 
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Pit  System  (ACASS)  Process 


The  Architect-engineer  Contract  Administration  Support  System  (ACASS)  is 
sponsored  by  the  Engineering  Division,  Directorate  of  Military  Programs  (CEMP-ES)  at 
Headquarters,  U.S.  Army  Corps  of  Engineers.  The  North  Pacific  Division,  U.S.  Army 
Corps  of  Engineers  (CENPD-CT)  is  responsible  for  the  day-to  day  management  of 
ACASS.  ACASS  provides  performance  history  and  is  used  for  only  architect  and 
engineering  service  contractors.  The  portion  of  the  ACASS  system  that  this  report  refers 
to  is  the  portion  used  to  provide  one  of  five  classifications  to  vendors  based  on  their 
performance.  The  possible  classifications  are  Excellent,  Above  Average,  Average,  Below 
Average,  and  Poor.  The  ACASS  database  itself  resides  on  a  mainfi:ame  computer  located 
at  the  CENPD-CT  command  in  Portland.  Access  to  the  database  is  provided  via  a  menu 
driven  interface  that  buyers  and  other  ACASS  data  users  connect  with  via  a  modem. 

All  firms  that  desire  to  perform  Architect-engineering  services  for  the  U.S.  Army 
Corps  of  Engineers  (US  ACE)  are  required  to  register  themselves  by  completion  of  SF  254 
Architect-Engineer  and  Related  Services  Questionnaire.  This  Questionnaire  allows 
contractors  to  describe  the  type  of  firm  they  are,  their  technical  expertise,  and  their 
specific  experience.  The  SF  254  must  be  updated  on  an  annual  basis.  Contractors  are 
notified  in  writing  whenever  a  classification  of  Below  Average  or  Poor  is  assigned  and 
may  challenge  the  classification. 


Architect-engineer  Contract  Administration  Support  System 

Major  Processes 


Each  of  the  major  process  activities  has  been  disaggregated  into  subcomponent  processes: 
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A-  Collect,  Generate,  Calculate,  and  Validate  Performance  Information 

Activity  A  of  the  major  ACASS  process  has  been  disaggregated  here.  This  activity 
pertains  to  the  collection  of  contractor  performance  data;  verifying  that  the  data  is  correct; 
classifying  contractor  performance  as  either  Excellent,  Above  Average,  Average,  Below 

Average,  or  Poor,  satisfying  contractor  challenges;  and  releasing  the  classifications  for  use 
by  buyers. 


A.  1  Elex  Record  -  Electronic  records  to  be  input  into  ACASS  database.  This  activity  is 
included  here  as  a  starting  place  for  this  process. 

A. 2  Receive  Elex  Data  -  Maintain  interactive  input  programs,  batch  import  programs,  and 
error  checking  algorithms.  This  activity  is  explored  in  D.3  “Maintain  Interfaces,”  but 
included  here  because  it  is  a  part  of  this  process. 

A.3  Hard  Record  -  paper  record  input  into  ACASS  database.  This  activity  is  included  here 
as  the  other  starting  place  for  this  process. 

A.4  Receive  Hard  Copy  Data  -  Receive,  review,  validate  and  correct  paper  reports. 

A.5  Valid  Electronic  Data  -  review  data  and  determine  its  validity. 

A. 6  Correct  Data  -  reimport  data,  reject  &  return  to  data  source,  manually  correct  data. 
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A.7  Enter  Data  -  import  electronic  data  and  key  in  hardcopy  data  into  ACASS  database. 

A.8  Calculate  Classifications  -  assign  ACASS  classification  to  contractors.  These 
calculations  are  provided  automatically  by  the  ACASS  database  software  to  the  ACASS 
system. 

A.9  Government  Review  -  government  personnel  review  classification,  if  classification  is 
Below  Average  or  Poor,  information  is  provided  to  contractors  for  their  review  and 
included  in  the  ACASS  database.  If  Excellent,  Above  Average,  or  Average,  the  data  is 
included  in  the  ACASS  database,  and  not  provided  to  contractors. 

A.  10  Contractor  Review  -  computed  negative  performance  classifications  is  provided  to 
Contractors,  contractors  may  challenge  the  classification.  Contractors  may  also  request 
positive  classification  information. 

A.l  1  Challenge  -  does  the  contractor  challenge  the  rating? 

A.  12  Resolve  challenge  -  work  with  contractor  and  users  to  resolve  any  rating  challenges. 

A.  13  Modify,  Authorize  Use  -  Update  ACASS  database  ratings  as  a  result  of  challenge 
resolution  and  authorize  use  of  the  data  for  contractor  evaluation  purposes. 
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B.  File,  Store,  Maintain,  and  F*rovide  Access  to  Performance  Information 

This  activity  pertains  to  storing  the  contractor  information  electronically  such  that 
buyers  can  access  contractor  historical  performance  information.  ACASS  provides 
interactive  access  when  users  dial  up  through  their  modems  and  then  ACASS  provides  a 
menu  driven  interface  so  that  buyers  can  navigate  the  system  to  get  the  information  they 
require. 


ACASS 

File,  Store,  Maintain  and 
Provide  Access  to 
Performance  Information 


B.  I  Buyer  Access  Data  via  LAN  -  system  operators  update  the  ACASS  database  monthly. 
Buyers  access  contractor  classification  information  interactively  by  dialing  up  via  modem 
when  making  buys. 
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C.  Retrieve,  Gather  and  Use  Contractor  Past  Performance  Information 

Activity  C  pertains  to  the  use  of  contractor  performance  information  by  buyers  as 
part  of  the  bu3nng  process.  Generally  for  the  ACASS  process,  performance  information  is 
used  as  the  fourth  criterion  after  technical  qualification,  experience,  and  available  capacity. 


ACASS 

Retrieve,  Gather  and  Use  Contractor  Past  Performance  Information 


C.I  Buyer  Reviews  Bids.  TTie  standard  criteria  in  order  of  importance  is  technically 
qualified,  experienced,  possesses  the  required  capacity  to  do  the  work,  has  the  best  past 
performance,  geographic  proximity,  participation  in  Small  Business  and/or  Small 
Disadvantaged  Business,  and  volume  of  DoD  contract  awards.  The  later  two  evaluation 
criteria  are  not  mentioned  in  the  above  diagram  because  they  are  criterion  that  are 
generally  used  after  the  Past  Performance  criterion  has  been  applied. 

C.2  A  Most  Qualified  Contractor?  -  if  the  buyer  determines  that  one  bidding  contractor  is 
the  most  technically  qualified,  an  award  is  made  to  that  contractor. 

C.3  A  Most  Experienced  Contractor?  -  if  two  or  more  contractors  are  equally  qualified, 
and  the  buyer  determines  that  one  bidding  contractor  is  the  most  experienced,  an  award  is 
made  to  that  contractor. 

C.4  A  Contractor  with  Required  Capacity?  -  if  two  or  more  contractors  are  equally 
experienced,  and  the  buyer  determines  that  one  bidding  contractor  has  the  required 
capacity,  an  award  is  made  to  that  contractor. 

C.5  A  Contractor  with  the  Best  Past  Performance?  -  if  two  or  more  contractors  possess 
the  required  capacity,  and  the  buyer  determines  that  one  bidding  contractor  has  the  best 
past  performance,  an  award  is  made  to  that  contractor. 

C.6  Make  Award  -  the  buyer  issues  an  award  to  the  selected  vendor. 
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D.  Design,  Operate  and  Manage  the  Automated  Elements  of  the  Process 

Activity  D  pertains  to  the  computer  and  communication  system  management 
System  operators  maintain  the  databases,  update  documentation,  upgrade  SW  &  HW  as 
new  versions  are  released,  maintain  and  upgrade  user  interfaces,  and  ensure  that  import 
and  export  functions  properly. 


ACASS 

Design,  Operate  and  Manage  the  Automated 
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D.  1  Maintain  guides  -  guides,  procedures,  handbooks  for  the  Architect-engineer  Contract 
Administration  Support  System  must  be  kept  up  to  date  and  current,  documenting  any 
changes  in  procedure,  responsibility,  and  data  elements. 

D.2  Manage  Database  -  system  operators  must  manage  the  database  to  ensure  that  is 
properly  updated  and  used. 

D.3  Maintain  Interfaces  -  system  administrators  must  maintain  the  automated  interfaces 
for  both  incoming  and  outgoing  data. 

D.4  Analyze  Database  -  system  operators  must  analyze  the  database  to  ensure  data 
elements  are  not  corrupted,  and  the  data  remains  normalized. 

D.5  Analyze/Publish  Statistics  -  operating  reports  and  statistics  are  kept  up  to  date  and  are 
published  periodically  and  on  an  adhoc  basis. 

D.6  Upgrade/Enhance  Operation  -  system  operators  must  upgrade  HW  &  SW  ensuring 
that  only  up  to  date  versions  of  both  Commercial-Off-The-Shelf  (COTS)  HW  &  SW  and 
any  internally  developed  code  is  being  used. 
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ACASS 

Design,  Operate  and  Manage  the  Automated 
Elements  of  the  Process 


D.7  Architect-engineer  Contract  Administration  Support  System  -  this  encompasses  the 
entire  system  that  provides  the  ACASS  classifications  and  their  use  (A3,  &  C). 
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E.  Manage  and  Oversee  the  Total  Past  Performance  Information  Process 

Activity  E  refers  to  the  administration  and  overall  management  of  the  Architect- 
engineer  Contract  Administration  Support  System.  Administrative  personnel  provide 
inforrnation  about  the  system,  provide  training,  oversee  system  operations,  and  perform 
administrative  tasks. 


ACASS 

Manage  and  Oversee  the  Total  Past 
Performance  Information  Process 


E.  1  Respond  to  Requests  -  system  administrators  respond  to  routine  and  special  requests 
for  information. 


E.2  Manage  Data  -  system  administrators  manage  the  information  produced  for  and  by  the 
ACASS  system. 

E.3  Manage  System  Ops  -  system  administrators  manage  the  day-to-day  operational 
activities. 


E.4  Admin  Functions  -  system  administrators  perform  necessary  administrative  activities. 

E.5  Respond  to  Questions  -  system  administrators  respond  to  questions  about  the  system, 
its  operation,  and  personnel. 

E.6  Provide  Training  -  system  administrators  provide  training  to  ACASS  users. 

E-7  Architect-engineer  Contract  Administration  Support  System  Process  -  this 
encompasses  the  entire  system  that  provides  the  ACASS  classifications,  their  use,  and  the 
information  systems  used  (A,B,C,  &  D). 
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Red/Yellow/Green  System  Overview 


Red/Yellow/Green  System  Past  Performance  Data  Acquisition 

PDREP  is  the  central  repository  for  contractor  past  performance  information. 


Red/Yellow/Green  System  PDREP  Access 
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Red/Yellow/Green  System  RYG  Application 


Red/Yellow/Green  System  RYG  Record  Layout 

The  RYG  evaluation  table  could  be  easily  imported  and  appended  to 
external  past  performance  databases. 


Red/Yellow/Green  System  Contractor  Access 

The  PDREP  system  is  being  etihanced  to  provide  contractors  with 
“push”  and  “puli”  access  to  their  profiles  and  RYG  classifications. 
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Red/Yellow/Green  System  Expansion  Issues 

Significant  expansion  of  the  PYG  past  performance  evaluation  system 
would  require  more  streamlined  data  acquisition,  improved  network 
access  capabilities,  and  greater  standardization  in  PC-based  software 
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Red/Yellow/Green  System  Expansion  Issues 
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Red/Yellow/Green  System  Expansion  Issues 

An  extrapolation  of  the  PDREP  system  from  the  current  number  of 
active  contractors  (1 1 ,000)  to  the  potential  number  of  contractors 
(700,000)  yields  the  following  (continued): 
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Automated  Best  Vaue  Model  (ABVM)  System  Past  Performance  Data  Acquisition 
Past  performance  data  for  ABVM  is  collected  from  a  variety  of  sources. 
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Automated  Best  Value  Model  (ABVM)  System  System  Components 
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Automated  Best  Value  Model  (ABVM)  System  Proposed  System  Upgrade 

A  new  system  architecture  for  ABVM  is  planned. 


Automated  Best  Value  Model  (ABVM)  System  Upgrade  Advantages 

The  proposed  ABVM  system  upgrades  will  yield  multiple  benefits  and 
advantages  over  the  existing  system  configuration. 
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